{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 深层神经网络\n",
    "前面一章我们简要介绍了神经网络的一些基本知识，同时也是示范了如何用神经网络构建一个复杂的非线性二分类器，更多的情况神经网络适合使用在更加复杂的情况，比如图像分类的问题，下面我们用深度学习的入门级数据集 MNIST 手写体分类来说明一下更深层神经网络的优良表现。\n",
    "\n",
    "## MNIST 数据集\n",
    "mnist 数据集是一个非常出名的数据集，基本上很多网络都将其作为一个测试的标准，其来自美国国家标准与技术研究所, National Institute of Standards and Technology (NIST)。 训练集 (training set) 由来自 250 个不同人手写的数字构成, 其中 50% 是高中学生, 50% 来自人口普查局 (the Census Bureau) 的工作人员，一共有 60000 张图片。 测试集(test set) 也是同样比例的手写数字数据，一共有 10000 张图片。\n",
    "\n",
    "每张图片大小是 28 x 28 的灰度图，如下\n",
    "\n",
    "![](https://ws3.sinaimg.cn/large/006tKfTcly1fmlx2wl5tqj30ge0au745.jpg)\n",
    "\n",
    "所以我们的任务就是给出一张图片，我们希望区别出其到底属于 0 到 9 这 10 个数字中的哪一个。\n",
    "\n",
    "## 多分类问题\n",
    "前面我们讲过二分类问题，现在处理的问题更加复杂，是一个 10 分类问题，统称为多分类问题，对于多分类问题而言，我们的 loss 函数使用一个更加复杂的函数，叫交叉熵。\n",
    "\n",
    "### softmax\n",
    "提到交叉熵，我们先讲一下 softmax 函数，前面我们见过了 sigmoid 函数，如下\n",
    "\n",
    "$$s(x) = \\frac{1}{1 + e^{-x}}$$\n",
    "\n",
    "可以将任何一个值转换到 0 ~ 1 之间，当然对于一个二分类问题，这样就足够了，因为对于二分类问题，如果不属于第一类，那么必定属于第二类，所以只需要用一个值来表示其属于其中一类概率，但是对于多分类问题，这样并不行，需要知道其属于每一类的概率，这个时候就需要 softmax 函数了。\n",
    "\n",
    "softmax 函数示例如下\n",
    "\n",
    "![](https://ws4.sinaimg.cn/large/006tKfTcly1fmlxtnfm4fj30ll0bnq3c.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于网络的输出 $z_1, z_2, \\cdots z_k$，我们首先对他们每个都取指数变成 $e^{z_1}, e^{z_2}, \\cdots, e^{z_k}$，那么每一项都除以他们的求和，也就是\n",
    "\n",
    "$$\n",
    "z_i \\rightarrow \\frac{e^{z_i}}{\\sum_{j=1}^{k} e^{z_j}}\n",
    "$$\n",
    "\n",
    "如果对经过 softmax 函数的所有项求和就等于 1，所以他们每一项都分别表示属于其中某一类的概率。\n",
    "\n",
    "## 交叉熵\n",
    "**交叉熵衡量两个分布相似性的一种度量方式**，前面讲的二分类问题的 loss 函数就是交叉熵的一种特殊情况，交叉熵的一般公式为\n",
    "\n",
    "$$\n",
    "cross\\_entropy(p, q) = E_{p}[-\\log q] = - \\frac{1}{m} \\sum_{x} p(x) \\log q(x)\n",
    "$$\n",
    "\n",
    "对于二分类问题我们可以写成\n",
    "\n",
    "$$\n",
    "-\\frac{1}{m} \\sum_{i=1}^m (y^{i} \\log sigmoid(x^{i}) + (1 - y^{i}) \\log (1 - sigmoid(x^{i}))\n",
    "$$\n",
    "\n",
    "这就是我们之前讲的二分类问题的 loss，当时我们并没有解释原因，只是给出了公式，然后解释了其合理性，现在我们给出了公式去证明这样取 loss 函数是合理的\n",
    "\n",
    "交叉熵是信息理论里面的内容，这里不再具体展开，更多的内容，可以看到下面的[链接](http://blog.csdn.net/rtygbwwwerr/article/details/50778098)\n",
    "\n",
    "下面我们直接用 mnist 举例，讲一讲深度神经网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torchvision.datasets import mnist # 导入 pytorch 内置的 mnist 数据\n",
    "\n",
    "from torch import nn\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 使用内置函数下载 mnist 数据集\n",
    "train_set = mnist.MNIST('./data', train=True, download=True)\n",
    "test_set = mnist.MNIST('./data', train=False, download=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看看其中的一个数据是什么样子的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60000"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset MNIST\n",
       "    Number of datapoints: 60000\n",
       "    Root location: ./data\n",
       "    Split: Train"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a_data, a_label = train_set[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABAElEQVR4nGNgGMyAWUhIqK5jvdSy/9/rGRgYGFhgEnJsVjYCwQwMDAxPJgV+vniQgYGBgREqZ7iXH8r6l/SV4dn7m8gmCt3++/fv37/Htn3/iMW+gDnZf/+e5WbQnoXNNXyMs/5GoQoxwVmf/n9kSGFiwAW49/11wynJoPzx4YIcRlyygR/+/i2XxCWru+vv32nSuGQFYv/83Y3b4p9/fzpAmSyoMnohpiwM1w5h06Q+5enfv39/bcMiJVF09+/fv39P+mFKiTtd/fv3799jgZiBJLT69t+/f/8eDuDEkDJf8+jv379/v7Ryo4qzMDAwMAQGMjBc3/y35wM2V1IfAABFF16Aa0wAOwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x25020865D30>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里的读入的数据是 PIL 库中的格式，我们可以非常方便地将其转换为 numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    }
   ],
   "source": [
    "a_data = np.array(a_data, dtype='float32')\n",
    "print(a_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们可以看到这种图片的大小是 28 x 28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   3.  18.\n",
      "   18.  18. 126. 136. 175.  26. 166. 255. 247. 127.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.  30.  36.  94. 154. 170. 253.\n",
      "  253. 253. 253. 253. 225. 172. 253. 242. 195.  64.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.  49. 238. 253. 253. 253. 253. 253.\n",
      "  253. 253. 253. 251.  93.  82.  82.  56.  39.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.  18. 219. 253. 253. 253. 253. 253.\n",
      "  198. 182. 247. 241.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.  80. 156. 107. 253. 253. 205.\n",
      "   11.   0.  43. 154.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.  14.   1. 154. 253.  90.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0. 139. 253. 190.\n",
      "    2.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.  11. 190. 253.\n",
      "   70.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.  35. 241.\n",
      "  225. 160. 108.   1.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.  81.\n",
      "  240. 253. 253. 119.  25.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "   45. 186. 253. 253. 150.  27.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.  16.  93. 252. 253. 187.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0. 249. 253. 249.  64.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "   46. 130. 183. 253. 253. 207.   2.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.  39. 148.\n",
      "  229. 253. 253. 253. 250. 182.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.  24. 114. 221. 253.\n",
      "  253. 253. 253. 201.  78.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.  23.  66. 213. 253. 253. 253.\n",
      "  253. 198.  81.   2.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.  18. 171. 219. 253. 253. 253. 253. 195.\n",
      "   80.   9.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.  55. 172. 226. 253. 253. 253. 253. 244. 133.  11.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0. 136. 253. 253. 253. 212. 135. 132.  16.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]\n",
      " [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.\n",
      "    0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]\n"
     ]
    }
   ],
   "source": [
    "print(a_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以将数组展示出来，里面的 0 就表示黑色，255 表示白色\n",
    "\n",
    "对于神经网络，我们第一层的输入就是 28 x 28 = 784，所以必须将得到的数据我们做一个变换，使用 reshape 将他们拉平成一个一维向量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def data_tf(x):\n",
    "    x = np.array(x, dtype='float32') / 255\n",
    "    x = (x - 0.5) / 0.5 # 标准化，这个技巧之后会讲到\n",
    "    x = x.reshape((-1,)) # 拉平\n",
    "    x = torch.from_numpy(x)\n",
    "    return x\n",
    "\n",
    "train_set = mnist.MNIST('./data', train=True, transform=data_tf, download=True) # 重新载入数据集，申明定义的数据变换\n",
    "test_set = mnist.MNIST('./data', train=False, transform=data_tf, download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([784])\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "a, a_label = train_set[0]\n",
    "print(a.shape)\n",
    "print(a_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "# 使用 pytorch 自带的 DataLoader 定义一个数据迭代器\n",
    "train_data = DataLoader(train_set, batch_size=64, shuffle=True)\n",
    "test_data = DataLoader(test_set, batch_size=128, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用这样的数据迭代器是非常有必要的，**如果数据量太大，就无法一次将他们全部读入内存，所以需要使用 python 迭代器，每次生成一个批次的数据**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a, a_label = next(iter(train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 784])\n",
      "torch.Size([64])\n"
     ]
    }
   ],
   "source": [
    "# 打印出一个批次的数据大小\n",
    "print(a.shape)\n",
    "print(a_label.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 使用 Sequential 定义 4 层神经网络\n",
    "net = nn.Sequential(\n",
    "    nn.Linear(784, 400),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(400, 200),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(200, 100),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(100, 10)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=784, out_features=400, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=400, out_features=200, bias=True)\n",
       "  (3): ReLU()\n",
       "  (4): Linear(in_features=200, out_features=100, bias=True)\n",
       "  (5): ReLU()\n",
       "  (6): Linear(in_features=100, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "交叉熵在 pytorch 中已经内置了，交叉熵的数值稳定性更差，所以内置的函数已经帮我们解决了这个问题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义 loss 函数\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(net.parameters(), 1e-1) # 使用随机梯度下降，学习率 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0, Train Loss: 0.509500, Train Acc: 0.837170, Eval Loss: 0.485445, Eval Acc: 0.842366\n",
      "epoch: 1, Train Loss: 0.171934, Train Acc: 0.946012, Eval Loss: 0.182450, Eval Acc: 0.940665\n",
      "epoch: 2, Train Loss: 0.116560, Train Acc: 0.963519, Eval Loss: 0.163148, Eval Acc: 0.945411\n",
      "epoch: 3, Train Loss: 0.090877, Train Acc: 0.971249, Eval Loss: 0.088837, Eval Acc: 0.970926\n",
      "epoch: 4, Train Loss: 0.072800, Train Acc: 0.977212, Eval Loss: 0.077864, Eval Acc: 0.975178\n",
      "epoch: 5, Train Loss: 0.064142, Train Acc: 0.979744, Eval Loss: 0.086222, Eval Acc: 0.974090\n",
      "epoch: 6, Train Loss: 0.052419, Train Acc: 0.983242, Eval Loss: 0.109800, Eval Acc: 0.967069\n",
      "epoch: 7, Train Loss: 0.043845, Train Acc: 0.986291, Eval Loss: 0.078314, Eval Acc: 0.975672\n",
      "epoch: 8, Train Loss: 0.052162, Train Acc: 0.984025, Eval Loss: 0.132545, Eval Acc: 0.964201\n",
      "epoch: 9, Train Loss: 0.037558, Train Acc: 0.987590, Eval Loss: 0.073922, Eval Acc: 0.978639\n",
      "epoch: 10, Train Loss: 0.040723, Train Acc: 0.987174, Eval Loss: 0.076172, Eval Acc: 0.977551\n",
      "epoch: 11, Train Loss: 0.028834, Train Acc: 0.990472, Eval Loss: 0.084344, Eval Acc: 0.975969\n",
      "epoch: 12, Train Loss: 0.025293, Train Acc: 0.991854, Eval Loss: 0.065087, Eval Acc: 0.981112\n",
      "epoch: 13, Train Loss: 0.025862, Train Acc: 0.992138, Eval Loss: 0.150969, Eval Acc: 0.957971\n",
      "epoch: 14, Train Loss: 0.018093, Train Acc: 0.994037, Eval Loss: 0.117743, Eval Acc: 0.970926\n",
      "epoch: 15, Train Loss: 0.015924, Train Acc: 0.995103, Eval Loss: 0.069340, Eval Acc: 0.981606\n",
      "epoch: 16, Train Loss: 0.013577, Train Acc: 0.995619, Eval Loss: 0.070877, Eval Acc: 0.982298\n",
      "epoch: 17, Train Loss: 0.011080, Train Acc: 0.996369, Eval Loss: 0.097533, Eval Acc: 0.976365\n",
      "epoch: 18, Train Loss: 0.009556, Train Acc: 0.997085, Eval Loss: 0.072429, Eval Acc: 0.981507\n",
      "epoch: 19, Train Loss: 0.008473, Train Acc: 0.997152, Eval Loss: 0.078409, Eval Acc: 0.980419\n"
     ]
    }
   ],
   "source": [
    "# 开始训练\n",
    "losses = []\n",
    "acces = []\n",
    "eval_losses = []\n",
    "eval_acces = []\n",
    "\n",
    "for e in range(20):\n",
    "    train_loss = 0\n",
    "    train_acc = 0\n",
    "    net.train()\n",
    "    for im, label in train_data:\n",
    "        im = Variable(im)\n",
    "        label = Variable(label)\n",
    "        # 前向传播\n",
    "        out = net(im)\n",
    "        loss = criterion(out, label)\n",
    "        # 反向传播\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        # 记录误差\n",
    "        train_loss += loss.item()\n",
    "        # 计算分类的准确率\n",
    "        _, pred = out.max(1)\n",
    "        num_correct = (pred == label).sum().item()\n",
    "        acc = num_correct / im.shape[0]\n",
    "        train_acc += acc\n",
    "        \n",
    "    losses.append(train_loss / len(train_data))\n",
    "    acces.append(train_acc / len(train_data))\n",
    "    # 在测试集上检验效果\n",
    "    eval_loss = 0\n",
    "    eval_acc = 0\n",
    "    net.eval() # 将模型改为预测模式\n",
    "    for im, label in test_data:\n",
    "        im = Variable(im)\n",
    "        label = Variable(label)\n",
    "        out = net(im)\n",
    "        loss = criterion(out, label)\n",
    "        # 记录误差\n",
    "        eval_loss += loss.item()\n",
    "        # 记录准确率\n",
    "        _, pred = out.max(1)\n",
    "        num_correct = (pred == label).sum().item()\n",
    "        acc = num_correct / im.shape[0]\n",
    "        eval_acc += acc\n",
    "        \n",
    "    eval_losses.append(eval_loss / len(test_data))\n",
    "    eval_acces.append(eval_acc / len(test_data))\n",
    "    print('epoch: {}, Train Loss: {:.6f}, Train Acc: {:.6f}, Eval Loss: {:.6f}, Eval Acc: {:.6f}'\n",
    "          .format(e, train_loss / len(train_data), train_acc / len(train_data), \n",
    "                     eval_loss / len(test_data), eval_acc / len(test_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画出 loss 曲线和 准确率曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x25027e8af70>]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgrklEQVR4nO3deZRc5Xnn8e9TVb1v1VK3kLpbICGEZMlIQBRsbFY7toE4FvZ4JnhJbAeHQyaQzZ4xmcx4nElmEttJDolNzCE28RLHODiBKLYwsWMwXoKjBi0gkEBIAnVray29r9X1zB91u1U03eqSVN1Vde/vc06dqrr3dtWjq9JPb7/13vc1d0dEREpfrNAFiIhIfijQRURCQoEuIhISCnQRkZBQoIuIhIQCXUQkJBToEilmdq+Z/a+z/NnHzeyj+a5JJF8ShS5AJFdmth/4qLt//2xfw91vz19FIsVFLXQJDTNTA0UiTYEuJcHMvgacD/yLmfWb2X83s2Vm5mZ2q5m9AvwgOPZBMztsZj1m9oSZrc16nS+b2R8Hj68zsw4z+5iZHTWzQ2b2kRzriZnZ/zSzl4Of/aqZNQT7Ks3s78zsuJl1m9kWMzsv2PdhM9trZn1mts/MPpDnUyURpkCXkuDuvwK8AvySu9e6+2eydl8LvA54R/D8EWAlsAh4Gvj6aV56MdAAtAK3AveYWWMOJX04uF0PXAjUAp8P9n0oeM2lwELgdmDIzGqAvwJudPc64E3AthzeSyQnCnQJg0+5+4C7DwG4+/3u3ufuI8CngPUTredpjAH/x93H3H0z0A+syuE9PwD8hbvvdfd+4PeBW4JunzEyQX6Ru4+7+1Pu3hv8XBp4vZlVufshd995tn9okakU6BIGByYemFnczP7UzF4ys15gf7CraYafPe7uqazng2Ra27NpAV7Oev4ymUEG5wFfAx4FHjCzg2b2GTMrc/cB4JfJtNgPmdl3zGx1Du8lkhMFupSSmaYGzd7+fmAj8Atkuj2WBdstz7UcBC7Ien4+kAKOBK39P3T3NWS6Vd4J/CqAuz/q7m8DlgC7gL/Jc10SYQp0KSVHyPRXn04dMAIcB6qB/zdHtXwD+F0zW25mtcH7fNPdU2Z2vZldYmZxoJdMF8y4mZ1nZu8K+tJHyHTvjM9RfRJBCnQpJX8C/M9g5MjHZzjmq2S6PzqB54An56iW+8l0rTwB7AOGgTuDfYuBb5EJ8+eBHwJ/R+bf28fItO5PkPky97/OUX0SQaYFLkREwkEtdBGRkFCgi4iEhAJdRCQkFOgiIiFRsMmMmpqafNmyZYV6exGRkvTUU08dc/fm6fYVLNCXLVtGe3t7od5eRKQkmdnLM+1Tl4uISEgo0EVEQkKBLiISEgp0EZGQUKCLiISEAl1EJCQU6CIiIVFygb7rcC+f+e4uegbHCl2KiEhRKblAf/n4IH/9+Eu8fGKg0KWIiBSVnALdzG4ws91mtsfM7ppm/3Vm1mNm24LbJ/NfakZrsgqAzpNDc/UWIiIladZL/4NltO4B3gZ0AFvMbJO7Pzfl0B+5+zvnoMZXaWsMAr1bgS4iki2XFvoVwB533+vuo8ADZBbhLYiGqjJqyuN0qIUuIvIquQR6K3Ag63lHsG2qK81su5k9YmZrp3shM7vNzNrNrL2rq+ssygUzoyVZpRa6iMgUuQS6TbNt6kKkTwMXuPt64HPAw9O9kLvf5+4b3H1Dc/O0sz/mpLWxioMKdBGRV8kl0DuApVnP28isWj7J3XvdvT94vBkoM7OmvFU5Rata6CIir5FLoG8BVprZcjMrB24BNmUfYGaLzcyCx1cEr3s838VOaG2sontwjIGR1Fy9hYhIyZl1lIu7p8zsDuBRIA7c7+47zez2YP+9wHuB3zCzFDAE3OLuU7tl8mZy6GL3EBefVzdXbyMiUlJyWrEo6EbZPGXbvVmPPw98Pr+lzSx7LLoCXUQko+SuFIVMlwtoLLqISLaSDPRFdZUkYqZAFxHJUpKBHo8ZS5KVuvxfRCRLSQY6QEuDhi6KiGQr2UBvbaxSC11EJEvJBnpbsoojfcOMjacLXYqISFEo2UBvbazCHQ73DBe6FBGRolC6gZ6sBtCsiyIigZIN9JZkJaCx6CIiE0o40LVykYhItpIN9MqyOE21FZpGV0QkULKBDsHQRQW6iAhQ4oHepnnRRUQmlXSgtyQr6eweIp2es5l6RURKRkkHemuyitFUmuMDo4UuRUSk4Eo70BszY9HV7SIiUuqBrqGLIiKTSjvQJxe6GCxwJSIihVfSgV5fmaC2IqEWuogIJR7oZkZrsorObk3QJSJS0oEOurhIRGRC6Qd6sorOk+pDFxEp/UBvrKJ3OEXf8FihSxERKaiSD/TJWRfV7SIiEVfygT4xFl2zLopI1JV8oLc16uIiEREIQaA311ZQFjc61EIXkYgr+UCPxYwlDVVqoYtI5JV8oEMwdFEtdBGJuHAEemOVvhQVkcjLKdDN7AYz221me8zsrtMc9/NmNm5m781fibNrTVZxtG+E0VR6Pt9WRKSozBroZhYH7gFuBNYA7zOzNTMc92ng0XwXOZvWxirc4VCPWukiEl25tNCvAPa4+153HwUeADZOc9ydwD8CR/NYX040L7qISG6B3gocyHreEWybZGatwLuBe0/3QmZ2m5m1m1l7V1fXmdY6c4FBoGvooohEWS6BbtNsm7oq893AJ9x9/HQv5O73ufsGd9/Q3NycY4mzW5KsBHS1qIhEWyKHYzqApVnP24CDU47ZADxgZgBNwE1mlnL3h/NR5GwqEnEW1VWoy0VEIi2XQN8CrDSz5UAncAvw/uwD3H35xGMz+zLw7fkK8wmaF11Eom7WLhd3TwF3kBm98jzwD+6+08xuN7Pb57rAXLXo4iIRibhcWui4+2Zg85Rt034B6u4fPveyzlxbsorv7TxCOu3EYtN1+4uIhFsorhSFTJfL6HiaY/0jhS5FRKQgwhPoGrooIhEXnkDXvOgiEnGhCXQtRSciUReaQK+vLKOuMqEWuohEVmgCHTL96LpaVESiKlSB3qaLi0QkwkIV6K1JLUUnItEVqkBvSVbRN5KiZ2is0KWIiMy7UAX6xNBF9aOLSBSFK9C10IWIRFi4Ar1RY9FFJLpCFehNNRWUJ2IKdBGJpFAFeixmtDRUqstFRCIpVIEOWuhCRKIrfIGuhS5EJKJCGOjVdPWNMDx22vWqRURCJ3SB3pKsBOBQz3CBKxERmV+hC3TNiy4iURW6QG9LVgO6WlREoid0gb64oRIzLUUnItETukAvT8Q4r05j0UUkekIX6JD5YrSze7DQZYiIzKtQBnprY7XGootI5IQz0JNVHO4ZZjzthS5FRGTehDPQG6sYG3e6+kYKXYqIyLwJZaC3TcyLrn50EYmQUAZ6SxDoHRrpIiIREspA10IXIhJFoQz02ooEDVVlulpURCIllIEOwTS66nIRkQjJKdDN7AYz221me8zsrmn2bzSzHWa2zczazeyq/Jd6ZrTQhYhEzayBbmZx4B7gRmAN8D4zWzPlsH8D1rv7pcCvAV/Mc51nbKKF7q6x6CISDbm00K8A9rj7XncfBR4ANmYf4O79fio5a4CCp2hrsoqB0XF6hsYKXYqIyLzIJdBbgQNZzzuCba9iZu82s13Ad8i00l/DzG4LumTau7q6zqbenGmki4hETS6BbtNse00L3N0fcvfVwM3AH033Qu5+n7tvcPcNzc3NZ1TomWpNaqELEYmWXAK9A1ia9bwNODjTwe7+BLDCzJrOsbZzoha6iERNLoG+BVhpZsvNrBy4BdiUfYCZXWRmFjy+HCgHjue72DOxsKacikRMLXQRiYzEbAe4e8rM7gAeBeLA/e6+08xuD/bfC/wn4FfNbAwYAn7ZCzy8xMxoTVZxsEeBLiLRMGugA7j7ZmDzlG33Zj3+NPDp/JZ27lobdXGRiERHaK8UhWAsuvrQRSQiQh/ox/pHGR4bL3QpIiJzLtSB3pLUSBcRiY5QB/rE0EXNuigiURDuQNfFRSISIaEO9MUNlcRMXS4iEg2hDvSyeIzz6ivVQheRSAh1oEOm26VDLXQRiYDwB3pjlb4UFZFICH+gJ6s43DPMeLrgU7SLiMyp8Ad6YxWptHOkd7jQpYiIzKnQB7ouLhKRqAh9oLdpLLqIREToA10LXYhIVIQ+0KvLEzRWlynQRST0Qh/ooHnRRSQaIhHoLQ2aF11Ewi8SgT7RQi/wqngiInMqGoGerGJobJzuwbFClyIiMmciEehtGukiIhEQiUBvTVYD0KEvRkUkxCIR6C3JSkAtdBEJt0gE+oKacirLYhq6KCKhFolANzNak5pGV0TCLRKBDtDaWK0uFxEJtegEelIXF4lIuEUo0Cs5MTDK4Giq0KWIiMyJ6AR6MBb9YLcWuhCRcIpOoAdj0dXtIiJhFZ1Ab9RCFyISbjkFupndYGa7zWyPmd01zf4PmNmO4PZTM1uf/1LPzXl1FcRjRmf3YKFLERGZE7MGupnFgXuAG4E1wPvMbM2Uw/YB17r7OuCPgPvyXei5SsRjLK6vVAtdREIrlxb6FcAed9/r7qPAA8DG7APc/afufjJ4+iTQlt8y8yNzcZG+FBWRcMol0FuBA1nPO4JtM7kVeGS6HWZ2m5m1m1l7V1dX7lXmSWujxqKLSHjlEug2zbZpV4ows+vJBPonptvv7ve5+wZ339Dc3Jx7lXnSmqzicO8wqfH0vL+3iMhcyyXQO4ClWc/bgINTDzKzdcAXgY3ufjw/5eVXS7KK8bRzuFfdLiISPrkE+hZgpZktN7Ny4BZgU/YBZnY+8E/Ar7j7C/kvMz80dFFEwiwx2wHunjKzO4BHgThwv7vvNLPbg/33Ap8EFgJ/bWYAKXffMHdln53WZHC1aI8CXUTCZ9ZAB3D3zcDmKdvuzXr8UeCj+S0t/yYCXS10EQmjyFwpClBVHmdhTblGuohIKEUq0CHzxajWFhWRMIpcoGtedBEJq+gFemNmKTr3aYfSi4iUrOgFerKK4bE0JwZGC12KiEheRS7QVyyqBWDT9tdcGyUiUtIiF+jXrGziLasX8Sebd7HzYE+hyxERyZvIBbqZ8dn3rqOxpow7v7GVgRGtMSoi4RC5QAdYWFvB3b98GfuODfC/N+0sdDkiInkRyUAHuHLFQu68/iK+9VQHD2/tLHQ5IiLnLLKBDvBbb13Jzy9r5A8eeob9xwYKXY6IyDmJdKAn4jH+8pbLSMRj3PGNpxlJjRe6JBGRsxbpQIfMVACffe86nu3s5TPf3V3ockREzlrkAx3g7WsX86ErL+BLP97HD3YdKXQ5IiJnRYEe+P2bXsfrltTz8Qd3cLhHKxqJSOlRoAcqy+J8/v2XMTw2zu98cyvjac31IiKlRYGeZUVzLX/4rrU8ufcE9zy2p9DliIicEQX6FO/9uTZuvrSFu7//Av+x70ShyxERyZkCfQoz44/ffQnnL6jmtx/YSvegZmUUkdKgQJ9GbUWCz73vco71j/DfvrVDc6eLSElQoM/gkrYGPnHDar733BG+9uTLhS5HRGRWCvTTuPWq5bxl9SL++NvPa6pdESl6CvTTmJhqN1mdmWp3cFRT7YpI8VKgz2JhbQV333JpZqrdf9ZUuyJSvBToOXjTiibuvP4iHtRUuyJSxBToOdJUuyJS7BToOcqeavfOb2yld3is0CWJiLyKAv0MTEy1u/NgD7/w5z/kkWcOaYy6iBQNBfoZevvaxTz8m2+mqbaC3/j60/z6V5/iYPdQocsSEVGgn411bUk23fFm/sdNq/nxni7e9hc/5Ms/2acZGkWkoBToZykRj3HbNSv43u9ey88tW8Cn/uU53vOFn/L8od5ClyYiEZVToJvZDWa228z2mNld0+xfbWb/bmYjZvbx/JdZvJYuqOYrH/l5/vKWS+k4Mcgvfe7HfPq7uxge0/qkIjK/Zg10M4sD9wA3AmuA95nZmimHnQB+C/izvFdYAsyMjZe28v3fu5Z3X9bKFx5/iXfc/QQ/fvFYoUsTkQjJpYV+BbDH3fe6+yjwALAx+wB3P+ruW4BIj+VrrCnns/95PX//628gZsYHv/Qzfu+b2zgxoCl4RWTu5RLorcCBrOcdwbYzZma3mVm7mbV3dXWdzUuUhDetaOKR376aO99yEZu2H+Stf/44//hUh4Y4isicyiXQbZptZ5VM7n6fu29w9w3Nzc1n8xIlo7Iszsfevorv/NbVLG+q4WMPbueDX/qZrjIVkTmTS6B3AEuznrcBB+emnPBZtbiOb93+Jv7o5tez40AP77j7Ce55bA9Do/rSVETyK5dA3wKsNLPlZlYO3AJsmtuywiUWM37ljRfwvd+7lutXLeKzj+7myj/9N/7s0d0c7R0udHkiEhKWS7+umd0E3A3Egfvd/f+a2e0A7n6vmS0G2oF6IA30A2vcfcZB2Rs2bPD29vZz/xOUoC37T/DFH+3lX587QiJmvGt9K7detZw1LfWFLk1EipyZPeXuG6bdV6gv6qIc6BNePj7A3/5kP//QfoDB0XHefNFCPnrVhVx7cTOx2HRfXYhI1CnQi1zP4Bjf2PIKX/7Jfg73DrOiuYZbr7qQ91zeSmVZvNDliUgRUaCXiLHxNJufOcTf/Ggvz3b2sqCmnA++4Xw+eOUFLKqrLHR5IlIEFOglxt35j30n+OKP9/H9549QFoux8dIWbr16OasXq59dJMpOF+iJ+S5GZmdmvOHChbzhwoXsOzbA3/5kHw+2d/DgUx1cvbKJj7x5GVde2ERVubpjROQUtdBLRPfgKH//H6/wlZ/u50jvCPGYsXpxHZcuTbJ+aZLLliZZ0VyrL1NFQk5dLiEymkrzoxe72PpKN9sOdLP9QDd9IykAaisSrGtreFXIL6pX37tImKjLJUTKEzHe+rrzeOvrzgMgnXb2HhuYDPdtB7q574m9pILFNloaKlm/NDkZ8pe0NlBTob92kTDSv+wSF4sZFy2q5aJFtbz359oAGB4bZ+fB3smA33agm0eePZw53mDV4nquXtnEdaua2XDBAsoTWudEJAzU5RIRJwZGJwO+/eUTbNl3ktHxNLUVCa66qInrVzdz3apFnKcuGpGipi4XYUFNOdevXsT1qxcBMDCS4qcvHeex3Ud5fNdRvrsz04Jfs6Se61Y1c/3qRVy2NEkirta7SKlQC11wd1440p8J991Had9/klTaqa9McM3FzVy/ahHXrmqmqbai0KWKRJ5GucgZ6R0e4ycvHuOx3Ud5bHcXXX0jmMG61gauW7WI61Y18/rWBsoK1HofSY1TkdAYfIkmBbqctXTaee5QL48H4b71lZOkHSoSMda21J8aQdOW5IKF1Zjldxx87/AYz3b2sKOjh2c6etje0U3HySHWL03y7ktbeOf6Fv3mIJGiQJe8OTkwyk9fOs62AyfZfqCHZzp7GBrLLNaRrC5jXVuSS9saWB8MkzyTsB0cTfHcwV62d/TwTEc3Ozp72Nt1aoWnpQuqWNea5PyF1fxwdxfPHeolHjOuWdnEzZe18vY1i3X1rISeAl3mTGo8zYtH+9l+oJvtHd1sO9DD7sO9BMPgaU1WBWPgG1jfluSStgaqyxOMpMbZdaiPHZ097DjQzTOdPbxwpG/y5xbXV3JJWwPrWhtYF4yfX1BT/qr33n24j4e2dvLP2zo51DNMTXmcd7x+Me+5rI0rVywkrqtmJYQU6DKvBkdT7DzYy7ZXutnWkbngqePkEJAZB790QTUHu4cYG8989hbUlLNuIryD0D+T4ZPptPOzfSd4eGsnm585RN9IikV1FWy8tIWbL2tlzZL6vHcFiRSKAl0K7lj/CDuCFvyLR/q4YGEN69sauKStgdZkVd4Cd3hsnB/sOspDWzt5fPdRxsadi8+r5ebLWtl4aSutyaqcXic1nqZ7aIyTA6OcGBjl5OAoJwbGODk4St9wiktaG7j64ibqK8vyUrdIrhToEkknB0b5zjOHeGhrJ0+9fBKANyxfwDvXLaE8EZsM6JOTgT3KycExTgyM0jM0NuPrJmJGKu0kYsaGZY28ZfUi3rJ6ESuaa/WbgMw5BbpE3ivHB3l4WycPb+1k77FTX7RWJGIsqCmnsbo8c19TzoLqssx99vbgPlldRiJmbD3QzQ92HeWxXUfZdbgPyHxp+5ZVmYu33njhQq02JXNCgS4ScHf2Hx+kPBGjsbqMqrL4ObeqO7uHeCwI95+8dIzhsTSVZTHevKKJ64PWe0uOXT0is1Ggi8yT4bFx/n3vcR7bdZQf7Do6+WXw6sV1k+GuKRXkXCjQRQrA3Xmpq58fBOGePaXCBQtrWFhbzsKaCprqymkK7hfWVLCwtpzm2goaa8oLdjWuFC9NziVSAGbGRYvquGhRHbdds4Le4TF+9MIxfrznGId6hjjeP8oLh/s41j/K6Hh62tdorC5jYW0FC2vKaaqroKmmnAU1FcQMxt0ZTzuptJMO7sfT021LM+5k7tNO2uGCBdWsba1nbUsDFzbV6DeGkFCgi8yT+soyfnHdEn5x3ZJXbXd3+kZSHO8f5Vj/CMf7Rzg2+fjU/fOHejnWN0LvcGryZ80yo27iMSNuwX3MiMdixGOQiMWytmWOcZwnXuhiJJX5T6QiEWP1knrWtkzcGli9uE5f6pYgBbpIgZkZ9ZVl1FeWsbypZtbjx4LWfNzsrNeQTY2nealrgJ0He9h5sJedB3v49vaD/P3PXsm8dsxY0VzD2pYG1rbUs6alnrVLGmio1rj7YqY+dBEBMr8pdJwcYufBXp6bDPpeDvcOTx7TmqxibUs9zXUV1FYmqKtIUFORoHbiVvnaxzXlCS1enkfqQxeRWZkZSxdUs3RBNTe8fvHk9mP9IzwXhPvOgz3sOtzHUy+fpG8kxWhq+r7/qWrK49RWZsK/Lgj7uooy6ioT1FWWUVuZoL4yMfm8LvjPoK6yLNheRmVZTBduzUKBLiKn1VRbwTUXN3PNxc2v2TeaSjMwkqJ/JEXfcIqB0RT9wyn6RlKZ7VMe949knvcPj3G0d4S+4RR9w2MMjI7PWkciZpn/CCoTk11U9VUToZ95nLkvO3VM9raK8P+moEAXkbNWnohRnshcYXsuxtMe/KcwFoR8iv6RzOPe4VPb+4dT9E4eM8b+Y4OTz/tHUrO+T11FgvqqMqrK41SVxaksi1FZFqeyLPN8clt5nMpEnKryOJWJWOa+7NStIhHL/NnjMSoSMcriwfOs7eXx2Lz/B6JAF5GCi8eMhqoyGqrO/kvX1Hia/pEUvUOZ0O8dHqN3KBP8vcMpeodObRseG2dobJyh0XH6R1J09Y0wkkozNJrZPjw2PjkK6FyUxe1U2GeF/vuvOJ+PXn3hOb/+VDkFupndAPwlEAe+6O5/OmW/BftvAgaBD7v703muVURkRol4jGR1Ocnqc/ttYUI67ZmQHzsV8kOjmfvRVJqR8TSjqVO3sfE0o8G2kYnt41n7gm0j4+k5W2Vr1kA3szhwD/A2oAPYYmab3P25rMNuBFYGtzcAXwjuRURKUixmma6ZEloFK5fLw64A9rj7XncfBR4ANk45ZiPwVc94Ekia2ZKpLyQiInMnl0BvBQ5kPe8Itp3pMZjZbWbWbmbtXV1dZ1qriIicRi6BPt3XtFOvRsrlGNz9Pnff4O4bmptfOwRKRETOXi6B3gEszXreBhw8i2NERGQO5RLoW4CVZrbczMqBW4BNU47ZBPyqZbwR6HH3Q3muVURETmPWUS7unjKzO4BHyQxbvN/dd5rZ7cH+e4HNZIYs7iEzbPEjc1eyiIhMJ6dx6O6+mUxoZ2+7N+uxA7+Z39JERORMaFZ7EZGQKNj0uWbWBbx8lj/eBBzLYzn5Vuz1QfHXqPrOjeo7N8Vc3wXuPu0wwYIF+rkws/aZ5gMuBsVeHxR/jarv3Ki+c1Ps9c1EXS4iIiGhQBcRCYlSDfT7Cl3ALIq9Pij+GlXfuVF956bY65tWSfahi4jIa5VqC11ERKZQoIuIhERRB7qZ3WBmu81sj5ndNc1+M7O/CvbvMLPL57G2pWb2mJk9b2Y7zey3pznmOjPrMbNtwe2T81Vf8P77zeyZ4L3bp9lfyPO3Kuu8bDOzXjP7nSnHzPv5M7P7zeyomT2btW2BmX3PzF4M7htn+NnTfl7nsL7Pmtmu4O/wITNLzvCzp/08zGF9nzKzzqy/x5tm+NlCnb9vZtW238y2zfCzc37+zpm7F+WNzLwxLwEXAuXAdmDNlGNuAh4hM33vG4GfzWN9S4DLg8d1wAvT1Hcd8O0CnsP9QNNp9hfs/E3zd32YzAUTBT1/wDXA5cCzWds+A9wVPL4L+PQMf4bTfl7nsL63A4ng8aenqy+Xz8Mc1vcp4OM5fAYKcv6m7P9z4JOFOn/neivmFnpRr5Tk7oc8WDfV3fuA55lmUY8iVywrTb0VeMndz/bK4bxx9yeAE1M2bwS+Ejz+CnDzND+ay+d1Tupz939194kl758kM311Qcxw/nJRsPM3IVgb+b8A38j3+86XYg70vK2UNNfMbBlwGfCzaXZfaWbbzewRM1s7v5XhwL+a2VNmdts0+4vi/JGZknmmf0SFPH8TzvNgOujgftE0xxTLufw1Mr91TWe2z8NcuiPoErp/hi6rYjh/VwNH3P3FGfYX8vzlpJgDPW8rJc0lM6sF/hH4HXfvnbL7aTLdCOuBzwEPz2dtwJvd/XIyi3j/ppldM2V/MZy/cuBdwIPT7C70+TsTxXAu/wBIAV+f4ZDZPg9z5QvACuBS4BCZbo2pCn7+gPdx+tZ5oc5fzoo50It+pSQzKyMT5l9393+aut/de929P3i8GSgzs6b5qs/dDwb3R4GHyPxam60YVpq6EXja3Y9M3VHo85flyERXVHB/dJpjCv1Z/BDwTuADHnT4TpXD52FOuPsRdx939zTwNzO8b6HPXwJ4D/DNmY4p1Pk7E8Uc6EW9UlLQ3/Yl4Hl3/4sZjlkcHIeZXUHmfB+fp/pqzKxu4jGZL86enXJYMaw0NWOrqJDnb4pNwIeCxx8C/nmaY3L5vM4JM7sB+ATwLncfnOGYXD4Pc1Vf9vcy757hfQt2/gK/AOxy947pdhby/J2RQn8re7obmVEYL5D59vsPgm23A7cHjw24J9j/DLBhHmu7isyvhDuAbcHtpin13QHsJPON/ZPAm+axvguD990e1FBU5y94/2oyAd2Qta2g54/Mfy6HgDEyrcZbgYXAvwEvBvcLgmNbgM2n+7zOU317yPQ/T3wO751a30yfh3mq72vB52sHmZBeUkznL9j+5YnPXdax837+zvWmS/9FREKimLtcRETkDCjQRURCQoEuIhISCnQRkZBQoIuIhIQCXUQkJBToIiIh8f8BLA5BU6t2NM4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title('train loss')\n",
    "plt.plot(np.arange(len(losses)), losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'train acc')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmeElEQVR4nO3de3hdV33m8e+ru2Vblm0pvkiOL8E4dkJCgjDh5maaFhxK8cAzl6QFWhMmzZRAmOmFDL3Qp+1MmZnSNi0ZPCkEJoWHtIUwhNZMoNBpQichdhLn4ksSx3ZiSbYjX2T5otuRfvPH3rIPx0fWcSzpSOe8n+c5j/bea23tdbaPXy+vs/faigjMzKx0VRS7AWZmNrEc9GZmJc5Bb2ZW4hz0ZmYlzkFvZlbiHPRmZiXOQW+WRdImSb9T7HaYjSf5OnorFZL2AR+NiH8odlvMphL36K1sSKoqdhvMisFBbyVB0l8BlwLfkXRS0m9KWiYpJN0i6RXgh2ndv5V0UNJxSQ9LuiLr93xF0h+my9dLapf0a5JelXRA0sbztGGjpJ2STkjaI+lXcso3SNomqUfSS5LWp9vnSfqypE5JxyT97/E/Q1bOHPRWEiLiQ8ArwM9HxKyI+G9ZxT8FrAbena5/F1gJXAI8CXztPL96ITAHaAFuAe6WNHeUuq8C7wUagI3An0q6FkDSWuA+4DeARmAdsC/d76+AeuCKtE1/Wsh7NiuU/ytr5eD3IuLUyEpE3DuyLOn3gGOS5kTE8Tz7DgK/HxEZYLOkk8Aq4LHcihHx91mr/yTpe8A7Sf4xuQW4NyK+n5Z3pMdfBNwIzI+IYyP7vra3aZafe/RWDvaPLEiqlPTZdOikh7O96qZR9j2ShvyI08CsfBUl3SjpMUlHJXUD78n6vUuAl/LstgQ4mhXyZuPOQW+lZLRLyLK3/wKwAfgZkiGZZel2XcyBJdUC3wT+GFgQEY3A5qzfux+4LM+u+4F5khov5vhm5+Ogt1JyCFgxRp3ZQD9whGRc/L+M07FrgFqgC8hIuhF4V1b5l4CNkm6QVCGpRdLlEXGA5DuD/yFprqRqSevGqU1mgIPeSssfAb8tqVvSr49S5z7gZZIx8h3kGWt/LSLiBPAJ4G+AYyT/c3gwq/xx0i9ogeMk4/BL0+IPkXwXsIvkC91PjkebzEb4hikzsxLnHr2ZWYlz0JuZlTgHvZlZiXPQm5mVuCl5Z2xTU1MsW7as2M0wM5s2nnjiicMR0ZyvbEoG/bJly9i6dWuxm2FmNm1Ienm0Mg/dmJmVOAe9mVmJGzPoJd2bzsX93CjlkvTnknZLemZkWta0bL2k59OyO8ez4WZmVphCevRfAdafp/xGkrm9VwK3Al+AZJZA4O60fA1ws6Q1F9NYMzO7cGMGfUQ8DBw9T5UNwH2ReAxoTOfYXgvsjog9ETEA3J/WNTOzSTQeY/QtZM33DbSn20bbnpekWyVtlbS1q6trHJplZmYwPkGfbx7vOM/2vCLinohoi4i25ua8l4KamdlrMB7X0beTPCVnRCvQSTI/d77tZmbT0tBwcLI/k7z6MpzsH6Snb2Q5Q9/gEMMBI7MCD0cQkfRwR5YhKU/qQZAsE0F9bRW3/VS+59NcnPEI+geB2yXdD7wFOB4RByR1ASslLSeZ+/smkjm6zcwmVURwemCIE30ZTvQNcqI/c3Y5/XmyL5OEdv/Z4D7Rn+FkWudkf4bTA0MT2s5LZtcWJ+glfR24HmiS1A58BqgGiIhNJI9Lew+wm+R5mhvTsoyk24GHgEqSByNvH/d3YGYlIyLozwzTOzBE72D6GvjJn32DQ5weyLM+OMTpMwGeoScroE/2ZxgaPv+zNySYWVPF7LrkNau2ijkzqmltnMGs2nRbuj0pr2ZWWnd2bVJWV1VJhQSCCoEklP7uCunMcSrObNeZehNpzKCPiJvHKA/gY6OUbSb5h8DMylB/ZoijpwY4cnKAo6eS15FTAxw91c+RkyPL6faT/Zzoz3Chz0KSoL66khk1yWt2bTWz66ponVtPQxrOSXhXn/1Zm7utipk1VVRUTGzgFsuUnOvGzKa+waFh2o/1svfwSfYePs3ewyc50N33E+F9sj+Td9/KCjFvZg3zZ9Ywb2YNVyxuYP7MGhpmVDOjpvJMcNdVVzIjXa7PWZ9RnazXVlVMeI94unPQm9mohoeDzuO97EuDfCTQ9x05zStHT//EcEhDXRUtc+tpmlXD0vn1WUFemyzPqjmzraGuumR7z1ORg96sxEQEmeEgMxQMDA2TGRpmcCgYHBpOX2eXM8PBYGaYwfTnkVP97Dl8in2HT7H38ClePnKa/szwmd89o7qSZU0zWbOogfe8YSHLm2axvKme5U2zmFtf7Z71FOWgN5sGIoKevgyHevo4eLyPg1k/Dx3v48DxPg719NHTN8jg0AUOcueoqazg0vn1LG+ayfWrLmF500yWzZ/J8qaZLGiodZhPQw56sykgIthz+BQvHjqZhHlOgB/s6ct7ad+8mTUsaKhjYUMtVy9pZM6MamoqRXVlBdVVFVRViJqqCqorzy5XVVRQXSmqqyqozllurK9mceMMKj2sUlIc9GZFEBG8cvQ0j750hEf3HOHRl47w6on+M+VVFWJBQx0LGmpZvaiB61ddwsI5tSycM4OFDXUsbKjjkoZa6qori/gubLpw0JtNko7u3iTYXzrCY3uO0NHdC0DTrFreetl83rpiPle2NLBwTh1NM2v9ZaWNGwe92QQ51NN3Jtgf3XOEV46eBmBufTXXrZjPr/zUCt66Yj6vu2SWx71tQjnorewd7x3k0ZcO88iLhzlycoDa6gpqqyqoraqkrjr5WVtVkW4/d1tdVeWZsn1HTp0J9j1dp4DkssO3rJjPL79tGW+9bD6rFsx2b90mlYPepoz+zBAi+cJwImWGhnmm4zgPv9DFIy8eZtv+boaGg1m1VSxurGMgM0zf4DD9mSH6M8NnJqoq1KzaKt68bC43vXkJb13RxJrFDf5y04rKQW9Fc+zUAFtfPsbWfUfZsu8oz3YcB2BF0yxWLZydvBYkP1saZ1xUL7j92GkeefEwj7zYxY9ePExPXwYJrmqZw69efxnvXNnMNZc2Ul2Z/x+ZzNDwmdDvzwynryH6B89u7xsconl2LW9omUPVKL/HrBgc9DYpIoL2Y71s2XeULfuOsWXfUXa/ehKA6kpxVWsjH3n7cioqxAsHT/DEy8d48Omzs1rPrKnk9Qtnc3ka/slyA/Nm1uQ93qn+DD/ee4SHXzjMwy92nRlGWdhQx/orF/LOlc2843VNzB1l/1xVlRVUVVYws9Z/ZWz68afWJsTQcLDrYA9b01Dfsu8oh3qSywdn11XRtnQu77+mhTcvm8dVrXPyXiZ4om+QFw6d4PmDJ3n+YA+7Dp7gu88d5OuPn31wWfPs2jO9/lULZ3P4ZD8Pv9DFEy8fY3AoqKuu4C3L5/OLb1nKupVN/uLTypKD3sbFQGaYbfu7eXzvER7fd4wnXz52ZkKrRXPqeMvy+bx52Vzals0r+MvI2XXVvGnpPN60dN6ZbRFB14l+dh08wfMHT7Dr4AleOHSCrz728plb9VcvauAjb1/Outc386alc32tuZU9B729JhHBS10n03Hvw/x4zxFOpXdurlowmw1vXMza5fNoWzaPlsYZ43ZcSVzSUMclDXWse/3ZR04ODSc3IM2sreSS2XXjdjyzUlBQ0EtaD9xF8gCRL0bEZ3PK5wL3ApcBfcBHIuK5tOw/AB8leZrWs8DGiOgbt3dgk+bwyX7+eXcS7D968TAHe5I/xmXz63n/tS2843XNXLdiHo31hY17j6fKCrG8aeakH9dsOijkCVOVwN3Az5I8H3aLpAcjYkdWtU8D2yLi/ZIuT+vfIKkF+ASwJiJ6Jf0NySMFvzLO78MmQN/gEI/vPcqP0nDfeaAHgMb6at5+WRPvWNnEO17XxJJ59UVuqZmdTyE9+rXA7ojYA5A+G3YDkB30a4A/AoiIXZKWSVqQdYwZkgaBevyA8ClreDjYcaAn6bHv7mLLvmMMZIaprhRvWjqX33j3Kt65sokrFs/xdeFm00ghQd8C7M9abyd5CHi2p4EPAD+StBZYCrRGxBOS/hh4BegFvhcR38t3EEm3ArcCXHrppRf0JuzCDQ0Hew+f5LmOHp7rOM5zncfZ3tnDib7kC9RVC2bzoeuW8o6VTbxl+Tzqa/x1jtl0Vcjf3nxdt9z7BD8L3CVpG8k4/FNAJh273wAsB7qBv5X0wYj46jm/MOIe4B6Atra2i5tQ237C4NAwu189yXMdSZg/23GcHZ099A4mX57WVFWwelED77t6MW9aOpd3vK6JSxr8haZZqSgk6NuBJVnrreQMv0RED7ARQMlFynvT17uBvRHRlZY9ALwNOCfobXz0DQ7xwqETSU+98zjbO46z8+AJBtJLD+trKrlicQP/9s1LuLJlDle2NHBZ86xR7wg1s+mvkKDfAqyUtBzoIPky9ReyK0hqBE5HxADJFTYPR0SPpFeA6yTVkwzd3ABsHcf2G8kwzPd3HOTeH+3jyVeOkUknZmmoq+LKljn88tuWccXiBq5smcOy+TM9vm5WZsYM+ojISLodeIjk8sp7I2K7pNvS8k3AauA+SUMkX9Lekpb9WNI3gCeBDMmQzj0T8k7KUN/gEA882cFfPrKHvYdPcem8em5dt4I3tMzhypY5tM6d4btAzQxFTL3h8La2tti61R3/0XSfHuCrj73MV/7fPg6fHOCq1jn8yrrLWH/lQvfWzcqUpCcioi1fmS+lmEY6unv50iN7uX/LK5weGOL6Vc38yrrLuG7FPPfczWxUDvppYEdnD/c8/BLfeeYAAt539WL+3boVrF7UUOymmdk04KCfoiKCR186wqaH9/DwC13MrKlk49uW8ZF3LGfxOM4dY2alz0E/xWSGhvnucwf5nw+/xHMdPTTNquU33r2KD75lKXPqq4vdPDObhhz0U0RmaJi/3rqfTf/0EvuP9rKiaSZ/9IE38P5rWjzNrpldFAf9FLB131F+59vb2Xmgh2subeS3f24NP7t6gR8gbWbjwkFfRK+e6OOz393FA092sHhOHf/jF6/lxisX+goaMxtXDvoiGBwa5r5HX+bPvv8CfZkhfvX6y7j9p1/nicPMbEI4WSbZY3uO8Jlvb+f5Qyf4qdc385mfX8OK5lnFbpaZlTAH/SQ51NPHf/77nTz4dCetc2dwz4fexM+uWeBhGjObcA76CTaQGebL/7yXP//BiwwOB5+4YSW/ev1lvpLGzCaNg34C/ejFw3zmwed4qesUP7P6En73vVdw6Xw/ds/MJpeDfgJ0dvfyh3+/g83PHmTp/Hru/eU2fvryBWPvaGY2ARz046g/M8QXH9nL53+4myD4tZ99Pf9u3QoP05hZUTnox8npgQz/6guPsuNAD+uvWMhvv3c1rXM9TGNmxeegHye/++3t7DzYw6YPXsv6KxcVuzlmZmcU9KBQSeslPS9pt6Q785TPlfQtSc9IelzSlVlljZK+IWmXpJ2S3jqeb2Aq+OYT7XzjiXY+/tMrHfJmNuWMGfSSKoG7gRuBNcDNktbkVPs0sC0irgI+DNyVVXYX8H8i4nLgamDneDR8qnip6yS/8+3neMvyedxxw8piN8fM7ByF9OjXArsjYk/68O/7gQ05ddYAPwCIiF3AMkkLJDUA64AvpWUDEdE9Xo0vtr7BIT72tSepq67krpuu8WP8zGxKKiToW4D9Wevt6bZsTwMfAJC0FlgKtAIrgC7gy5KekvRFSTPzHUTSrZK2Stra1dV1gW+jOP7w73ew6+AJPvevr2bhnLpiN8fMLK9Cgj5fNzX3ieKfBeZK2gZ8HHgKyJB82Xst8IWIuAY4BZwzxg8QEfdERFtEtDU3NxfY/OLZ/OwBvvrYK9y6bgX/4vJLit0cM7NRFXLVTTuwJGu9FejMrhARPcBGACWTt+xNX/VAe0T8OK36DUYJ+ulk/9HTfOobz/DGJY38+rtWFbs5ZmbnVUiPfguwUtJySTXATcCD2RXSK2tq0tWPAg9HRE9EHAT2SxpJwxuAHePU9qIYyAxz+9efAsFf3HwNNVUFXbhkZlY0Y/boIyIj6XbgIaASuDcitku6LS3fBKwG7pM0RBLkt2T9io8DX0v/IdhD2vOfrv77Q7t4en83X/jFa1kyzzdEmdnUV9ANUxGxGdics21T1vKjQN5rCyNiG9D22ps4dfxw1yH+8pG9fOi6pdz4Bl8vb2bTg8cdCnTgeC+/9jdPs3pRA7/1c6uL3Rwzs4I56AuQGRrmjq9voz8zzN2/cI0nKTOzacVz3RTgz3/wIo/vO8qf/tur/dg/M5t23KMfwz/vPsxf/ONu/tWbWnn/Na3Fbo6Z2QVz0J9H14l+PvnX21jRNJPf33BFsZtjZvaaeOhmFMPDwX/8m2309A7yV7espb7Gp8rMpif36EfxhX96iUdePMxnfv4KLl/YUOzmmJm9Zg76PLbuO8qffP8F3nvVIm5eu2TsHczMpjAHfY7u0wN84utP0dI4gz/6wBtIpu4xM5u+PPCcJSL49b99hq6T/Xzz37+N2XXVxW6SmdlFc48+y5f/eR//sPMQd964mqtaG4vdHDOzceGgT0UEf/oPL7Du9c185O3Lit0cM7Nx46BPdZ8e5ERfhnUrmzwub2YlxUGf6ujuBaB17owit8TMbHw56FMjQb+40UFvZqXFQZ/qTIO+xUFvZiWmoKCXtF7S85J2Szrnma+S5kr6lqRnJD0u6cqc8kpJT0n6u/Fq+HjrONZLXXUF82bWjF3ZzGwaGTPoJVUCdwM3AmuAmyWtyan2aWBbRFwFfBi4K6f8DmDnxTd34nQe72Vx4wx/EWtmJaeQHv1aYHdE7ImIAeB+YENOnTXADwAiYhewTNICAEmtwM8BXxy3Vk+AjmO9HrYxs5JUSNC3APuz1tvTbdmeBj4AIGktsBQYmbz9z4DfBIbPdxBJt0raKmlrV1dXAc0aXx3dfQ56MytJhQR9vrGMyFn/LDBX0jbg48BTQEbSe4FXI+KJsQ4SEfdERFtEtDU3NxfQrPHTNzjE4ZP9vuLGzEpSIXPdtAPZUzi2Ap3ZFSKiB9gIoGSQe2/6ugl4n6T3AHVAg6SvRsQHx6Ht4+bA8T7AV9yYWWkqpEe/BVgpabmkGpLwfjC7gqTGtAzgo8DDEdETEf8pIlojYlm63w+nWshDMj4PvobezErTmD36iMhIuh14CKgE7o2I7ZJuS8s3AauB+yQNATuAWyawzeOu03fFmlkJK2ia4ojYDGzO2bYpa/lRYOUYv+P/Av/3gls4Cdq7e5FgQUNdsZtiZjbufGcsSY9+wew6aqp8Osys9DjZSMboFze6N29mpclBT3JXbMvc+mI3w8xsQpR90A8PBwe6+9yjN7OSVfZBf/hkPwNDw7T60kozK1FlH/TtnofezEpc2Qf9mXnofQ29mZWosg963xVrZqWu7IO+s7uX2XVVNNRVF7spZmYTouyDvqPb89CbWWlz0Hf3edjGzEqag/7YaffozayklXXQn+gbpKcv4x69mZW0sg76zu70gSO+tNLMSliZB316Db2nPzCzElZQ0EtaL+l5Sbsl3ZmnfK6kb0l6RtLjkq5Mty+R9I+SdkraLumO8X4DF6P9TNB7QjMzK11jBr2kSuBu4EZgDXCzpDU51T4NbIuIq4APA3el2zPAr0XEauA64GN59i2azu5eqipE8+zaYjfFzGzCFNKjXwvsjog9ETEA3A9syKmzBvgBQETsApZJWhARByLiyXT7CWAn0DJurb9IHcd6WdRYR2WFit0UM7MJU0jQtwD7s9bbOTesnwY+ACBpLbAUaM2uIGkZcA3w49fY1nHX2d3L4jn+ItbMSlshQZ+vuxs5658F5kraBnwceIpk2Cb5BdIs4JvAJyOiJ+9BpFslbZW0taurq5C2X7SO7l5fcWNmJa+Qh4O3A0uy1luBzuwKaXhvBJAkYG/6QlI1Sch/LSIeGO0gEXEPcA9AW1tb7j8k425waJhDPX2+WcrMSl4hPfotwEpJyyXVADcBD2ZXkNSYlgF8FHg4InrS0P8SsDMi/mQ8G36xDh7vYzhw0JtZyRuzRx8RGUm3Aw8BlcC9EbFd0m1p+SZgNXCfpCFgB3BLuvvbgQ8Bz6bDOgCfjojN4/s2LlynHzhiZmWikKEb0mDenLNtU9byo8DKPPv9iPxj/EXX4QeOmFmZKNs7Y8/06H3VjZmVuLIN+o7uXubPrGFGTWWxm2JmNqHKOOg9D72ZlYfyDXrPQ29mZaIsgz4i6HSP3szKRFkG/bHTg/QODvmKGzMrC2UZ9J6H3szKSVkGfYfnoTezMlKeQX9s5K5Y9+jNrPSVZdB3dvdSV13BvJk1Y1c2M5vmyjLoO7p7Wdw4g2TONTOz0laWQd/Z3etr6M2sbJRl0Hc46M2sjJRd0PcNDnH45ICD3szKRtkFveehN7NyU4ZB3wd4HnozKx8FBb2k9ZKel7Rb0p15yudK+pakZyQ9LunKQvedbB3dpwE/QtDMyseYQS+pErgbuBFYA9wsaU1OtU8D2yLiKuDDwF0XsO+k6ujuQ4KFc3yzlJmVh0J69GuB3RGxJyIGgPuBDTl11gA/AIiIXcAySQsK3HdSdRzrZcHsOqory27UyszKVCFp1wLsz1pvT7dlexr4AICktcBSoLXAfUn3u1XSVklbu7q6Cmv9a9DZ3evxeTMrK4UEfb7bRyNn/bPAXEnbgI8DTwGZAvdNNkbcExFtEdHW3NxcQLNem5G7Ys3MykVVAXXagSVZ661AZ3aFiOgBNgIomVdgb/qqH2vfyTQ8HBw43st73rCoWE0wM5t0hfTotwArJS2XVAPcBDyYXUFSY1oG8FHg4TT8x9x3MnWd7GdwKDwPvZmVlTF79BGRkXQ78BBQCdwbEdsl3ZaWbwJWA/dJGgJ2ALecb9+JeStjOzMPvcfozayMFDJ0Q0RsBjbnbNuUtfwosLLQfYvl7Dz0DnozKx9ldY3h2UcIOujNrHyUVdB3dPcyu66K2XXVxW6KmdmkKaug9zz0ZlaOyiro24856M2s/JRV0PuuWDMrR2UT9Cf6Bunpy/iKGzMrO2UT9GfmoXfQm1mZKZugH5mH3j16Mys3ZRT0SY++1WP0ZlZmyifoj/VSXSmaZ9UWuylmZpOqbIK+s7uXRXNmUFGRb+ZkM7PSVTZBn8xD71krzaz8lE3QJ3fF1he7GWZmk64sgn5waJhDPX2eh97MylJZBP3B430Mh+ehN7PyVBZBP/LAEV9Db2blqKCgl7Re0vOSdku6M0/5HEnfkfS0pO2SNmaV/Yd023OSvi5p0sdPPA+9mZWzMYNeUiVwN3AjsAa4WdKanGofA3ZExNXA9cDnJNVIagE+AbRFxJUkjxO8aRzbXxA/WcrMylkhPfq1wO6I2BMRA8D9wIacOgHMliRgFnAUyKRlVcAMSVVAPdA5Li2/AJ3He2maVUNddeVkH9rMrOgKCfoWYH/Wenu6LdvnSR4Q3gk8C9wREcMR0QH8MfAKcAA4HhHfy3cQSbdK2ippa1dX1wW+jfNrP9br3ryZla1Cgj7fraSRs/5uYBuwGHgj8HlJDZLmkvT+l6dlMyV9MN9BIuKeiGiLiLbm5uYCm18YP1nKzMpZIUHfDizJWm/l3OGXjcADkdgN7AUuB34G2BsRXRExCDwAvO3im124iEjvinXQm1l5KiTotwArJS2XVEPyZeqDOXVeAW4AkLQAWAXsSbdfJ6k+Hb+/Adg5Xo0vxLHTg/QNDrtHb2Zlq2qsChGRkXQ78BDJVTP3RsR2Sbel5ZuAPwC+IulZkqGeT0XEYeCwpG8AT5J8OfsUcM/EvJX8fMWNmZW7MYMeICI2A5tztm3KWu4E3jXKvp8BPnMRbbwoIzdLeR56MytXJX9nrO+KNbNyV/JB39ndy4zqSubWVxe7KWZmRVEWQb+4sY7ku2Azs/JT8kHf0d1Ly1zPQ29m5avkgz65Wcrz0JtZ+SrpoO8bHOLwyQFfQ29mZa2kg77TV9yYmZV20Hd4Hnozs9IOevfozcxKPOg7jvVSIVg4x1/Gmln5Ku2g7+5jQUMd1ZUl/TbNzM6rpBOwo/u0x+fNrOyVdNB3dvd5fN7Myl7JBv3wcHDgeC8tnrXSzMpcyQZ918l+BofCPXozK3slG/Tt6QNHWh30ZlbmCgp6SeslPS9pt6Q785TPkfQdSU9L2i5pY1ZZo6RvSNolaaekt47nGxiNr6E3M0uMGfSSKoG7gRuBNcDNktbkVPsYsCMirgauBz6XPl8W4C7g/0TE5cDVTNIzY88+cMTX0JtZeSukR78W2B0ReyJiALgf2JBTJ4DZ6QPAZwFHgYykBmAd8CWAiBiIiO7xavz5dHb30lBXxew6P3DEzMpbIUHfAuzPWm9Pt2X7PLAa6ASeBe6IiGFgBdAFfFnSU5K+KGlmvoNIulXSVklbu7q6LvR9nKPjmOehNzODwoI+36OZImf93cA2YDHwRuDzaW++CrgW+EJEXAOcAs4Z4weIiHsioi0i2pqbmwtr/Xl0eB56MzOgsKBvB5ZkrbeS9NyzbQQeiMRuYC9webpve0T8OK33DZLgn3BJ0PuLWDOzQoJ+C7BS0vL0C9abgAdz6rwC3AAgaQGwCtgTEQeB/ZJWpfVuAHaMS8vPo6dvkBN9GV9xY2ZGMrRyXhGRkXQ78BBQCdwbEdsl3ZaWbwL+APiKpGdJhno+FRGH01/xceBr6T8Se0h6/xNq5NJK3xVrZlZA0ANExGZgc862TVnLncC7Rtl3G9D22pt44XwNvZnZWSV5Z2yH74o1MzujNIO+u4+aygqaZtUWuylmZkVXokHfy6LGOioq8l0ZamZWXkoy6Du7e1k8x8M2ZmZQokGf3BXroDczgxIM+sGhYQ6d8JOlzMxGlFzQHzzeR4SvuDEzG1FyQd/ha+jNzH5C6QX9Md8Va2aWreSCfuSu2EVzPHOlmRmUYNB3dPfSNKuWuurKYjfFzGxKKMmg9zz0ZmZnlWbQe3zezOyMkgr6iPBdsWZmOUoq6I+eGqBvcNg9ejOzLAUFvaT1kp6XtFvSOc98lTRH0nckPS1pu6SNOeWV6cPB/268Gp5PZ3cf4GvozcyyjRn0kiqBu4EbgTXAzZLW5FT7GLAjIq4Grgc+lz5RasQdwM5xafF5dHSfBvCzYs3MshTSo18L7I6IPRExANwPbMipE8BsSQJmAUeBDICkVuDngC+OW6tH0ZH26B30ZmZnFRL0LcD+rPX2dFu2zwOrgU7gWeCOiBhOy/4M+E1gmPOQdKukrZK2dnV1FdCsc3Uc66W+ppLG+urXtL+ZWSkqJOjzPb0jctbfDWwDFgNvBD4vqUHSe4FXI+KJsQ4SEfdERFtEtDU3NxfQrHN1dveyuHEGyX8szMwMCgv6dmBJ1norSc8920bggUjsBvYClwNvB94naR/JkM9PS/rqRbd6FMnNUh62MTPLVkjQbwFWSlqefsF6E/BgTp1XgBsAJC0AVgF7IuI/RURrRCxL9/thRHxw3FqfY6RHb2ZmZ1WNVSEiMpJuBx4CKoF7I2K7pNvS8k3AHwBfkfQsyVDPpyLi8AS2+xzDw8G61zezdvncyTysmdmUp4jc4fbia2tri61btxa7GWZm04akJyKiLV9ZSd0Za2Zm53LQm5mVOAe9mVmJc9CbmZU4B72ZWYlz0JuZlTgHvZlZiXPQm5mVuCl5w5SkLuDl17h7EzCpd+VeILfv4rh9F8ftuzhTuX1LIyLvjJBTMugvhqSto90dNhW4fRfH7bs4bt/FmertG42HbszMSpyD3sysxJVi0N9T7AaMwe27OG7fxXH7Ls5Ub19eJTdGb2ZmP6kUe/RmZpbFQW9mVuKmZdBLWi/peUm7Jd2Zp1yS/jwtf0bStZPcviWS/lHSTknbJd2Rp871ko5L2pa+fneS27hP0rPpsc95yksxz6GkVVnnZZukHkmfzKkzqedP0r2SXpX0XNa2eZK+L+nF9Gfex5uN9XmdwPb9d0m70j+/b0lqHGXf834WJrB9vyepI+vP8D2j7Fus8/fXWW3bJ2nbKPtO+Pm7aBExrV4kjzN8CVgB1ABPA2ty6rwH+C7JYw2vA348yW1cBFybLs8GXsjTxuuBvyviedwHNJ2nvKjnMOfP+yDJzSBFO3/AOuBa4Lmsbf8NuDNdvhP4r6O0/7yf1wls37uAqnT5v+ZrXyGfhQls3+8Bv17An39Rzl9O+eeA3y3W+bvY13Ts0a8FdkfEnogYAO4HNuTU2QDcF4nHgEZJiyargRFxICKeTJdPADuBlsk6/jgp6jnMcgPwUkS81julx0VEPAwczdm8Afhf6fL/Av5lnl0L+bxOSPsi4nsRkUlXHwNax/u4hRrl/BWiaOdvhCQB/wb4+ngfd7JMx6BvAfZnrbdzbogWUmdSSFoGXAP8OE/xWyU9Lem7kq6Y3JYRwPckPSHp1jzlU+Uc3sTof8GKef4AFkTEAUj+cQcuyVNnqpzHj5D8Dy2fsT4LE+n2dGjp3lGGvqbC+XsncCgiXhylvJjnryDTMeiVZ1vuNaKF1JlwkmYB3wQ+GRE9OcVPkgxHXA38BfC/J7l5b4+Ia4EbgY9JWpdTXvRzKKkGeB/wt3mKi33+CjUVzuNvARnga6NUGeuzMFG+AFwGvBE4QDI8kqvo5w+4mfP35ot1/go2HYO+HViStd4KdL6GOhNKUjVJyH8tIh7ILY+Inog4mS5vBqolNU1W+yKiM/35KvAtkv8iZyv6OST5i/NkRBzKLSj2+UsdGhnOSn++mqdOUc+jpF8C3gv8YqQDyrkK+CxMiIg4FBFDETEM/OUoxy32+asCPgD89Wh1inX+LsR0DPotwEpJy9Me303Agzl1HgQ+nF45ch1wfOS/2JMhHdP7ErAzIv5klDoL03pIWkvyZ3Fkkto3U9LskWWSL+2ey6lW1HOYGrUnVczzl+VB4JfS5V8Cvp2nTiGf1wkhaT3wKeB9EXF6lDqFfBYmqn3Z3/m8f5TjFu38pX4G2BUR7fkKi3n+Lkixvw1+LS+SK0JeIPk2/rfSbbcBt6XLAu5Oy58F2ia5fe8g+e/lM8C29PWenDbeDmwnuYrgMeBtk9i+Felxn07bMBXPYT1JcM/J2la080fyD84BYJCkl3kLMB/4AfBi+nNeWncxsPl8n9dJat9ukvHtkc/gptz2jfZZmKT2/VX62XqGJLwXTaXzl27/yshnLqvupJ+/i315CgQzsxI3HYduzMzsAjjozcxKnIPezKzEOejNzEqcg97MrMQ56M3MSpyD3sysxP1/kjmzV309ohcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(len(acces)), acces)\n",
    "plt.title('train acc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1136d2860>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEICAYAAABYoZ8gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcnHWV6P/P6ep9rU66O0svWTskIaST0GIEAgrRAQZFR/0JCopXxAgIqPwcXK46rxnv3HG8XkdFIpuigogIihqHmXFECEtISDoJIWTpzlLdWbo6ve/buX9UVadSqU5Xd9de5/169StVz1LPt59Un/rW9znP+YqqYowxJnWkxboBxhhjossCvzHGpBgL/MYYk2Is8BtjTIqxwG+MMSnGAr8xxqQYC/zGTJGIPC8it8S6HcZMlgV+k1RE5LCIrA/D69wsIpvD0SZj4o0FfmOMSTEW+E3SEJGfA1XA70WkW0S+5F2+VkReFpF2EdkpIu/02+dmEWkQkS4ROSQiHxORZcBG4B3e12kP4dhpIvI1ETkiIs0i8jMRKfKuyxaRX4jIKW8btorIrPGOH4FTY8wZLPCbpKGqNwFHgfeqar6qfltEyoE/Av8EzADuAX4jIqUikgd8H7haVQuAi4E6Vd0LbABe8b6OM4TD3+z9eRewEMgHfuhd9wmgCKgEZnpfu2+840/zNBgzIQv8JtndCGxS1U2qOqqq/wlsA67xrh8FVohIjqoeV9U9UzzOx4DvqmqDqnYDXwauF5F0YAhPwF+sqiOq+rqqdob5+MaEzAK/SXbzgA97h1javcM2lwJzVLUH+AieHvhxEfmjiCyd4nHmAkf8nh8B0oFZwM+B54AnROSYiHxbRDLCfHxjQmaB3ySbwHKzLuDnqur0+8lT1f8NoKrPqeq7gTnAW8CD47zORI7h+ZDxqQKGgZOqOqSq/6Cqy/EM51wLfHyC4xsTMRb4TbI5iWeM3ecXwHtF5G9ExOG90PpOEakQkVkicp13rH0A6MYz9OJ7nQoRyQzxuL8EPi8iC0QkH/hfwK9UdVhE3iUiF4iIA+jEM/QzOsHxjYkYC/wm2fwz8DXvsM49quoCrgO+ArjxfAP4//G899OAL+DprbcClwOf9b7OfwN7gBMi0hLCcR/BM6TzAnAI6Ac+5103G3gKT9DfC/zVu+25jm9MxIhNxGKMManFevzGGJNiLPAbY0yKscBvjDEpJqTALyJXicg+ETkoIvcGWf8xEdklIru9t8bXeJdXishfRORNEdkjIneF+xcwxhgzORNe3PWmoO0H3g00AluBG1T1Tb9tLgb2qmqbiFwNfFNV3y4ic/DcKLNdRAqA14H3++8bTElJic6fP386v5cxxqSU119/vUVVS0PZNj2EbS4CDqpqA4CIPIEnPW4seKvqy37bvwpUeJcfB457H3eJyF6g3H/fYObPn8+2bdtCab8xxhhARI5MvJVHKEM95Xhyn30avcvG8yngT0EaNR9YDWwJtpOI3Coi20Rkm9vtDqFZxhhjpiKsF3dF5F14Av/fByzPB34D3O1XnOoMqvqAqtaqam1paUjfVowxxkxBKEM9TXjKyfpUeJedQURWAg/hKTF7ym95Bp6g/5iqPj295hpjjJmuUHr8W4Fqbw2STOB64Fn/DUSkCngauElV9/stF+BhPBd+vxu+ZhtjjJmqCXv83iJTd+ApK+sAHlHVPSKywbt+I/B1PPXGf+SJ9Qyrai1wCXATsFtEfBNMfEVVN4X/VzHGGBOKuKzVU1tbq5bVY4wxoROR170d7gnZnbvGGJNikibwDw6Pcv/z9byw31JBjTHmXJIm8Gc4hAdeqOcPu47FuinGGBPXkibwiwg1lU52ujpi3RRjjIlrSRP4AWoqnBxo7qJnYDjWTTHGmLiVXIG/sohRhTearNdvjDHjSarAv7LCCcDOxvYYt8QYY+JXUgX+kvwsKopzbJzfGGPOIakCP+C5wGs9fmOMGVfyBf6KIhrb+mjpHoh1U4wxJi4lYeD3jPPvsl6/McYElXSBf0V5EWkCdTbOb4wxQSVd4M/LSqe6rMB6/MYYM46kC/zgyeff6WonHiuPGmNMrCVp4HfS1juEq7Uv1k0xxpi4k5yB33uBt86Ge4wx5ixJGfjPm11AVnoau1wW+I0xJlBSBv4MRxrnzy20G7mMMSaIkAK/iFwlIvtE5KCI3Btk/cdEZJeI7BaRl0WkJtR9I6Wm0snupg6GR0ajdUhjjEkIEwZ+EXEA9wFXA8uBG0RkecBmh4DLVfUC4B+BByaxb0SsqnTSPzTKgebuaBzOGGMSRig9/ouAg6raoKqDwBPAdf4bqOrLqtrmffoqUBHqvpEyVqnTxvmNMeYMoQT+csDl97zRu2w8nwL+NNl9ReRWEdkmItvc7unPmzt/Zi6F2ek2zm+MMQHCenFXRN6FJ/D//WT3VdUHVLVWVWtLS0vD0RZqKp1WusEYYwKEEvibgEq/5xXeZWcQkZXAQ8B1qnpqMvtGSk2Fk/0nu+gbHInWIY0xJu6FEvi3AtUiskBEMoHrgWf9NxCRKuBp4CZV3T+ZfSOpptLJyKiy55j1+o0xxmfCwK+qw8AdwHPAXuBJVd0jIhtEZIN3s68DM4EfiUidiGw7174R+D2CqqkoAqDOLvAaY8yY9FA2UtVNwKaAZRv9Ht8C3BLqvtFSVpjN3KJsdjVaj98YY3yS8s5dfysrbCpGY4zxl/SBv6bSyZFTvbT1DMa6KcYYExdSIPB7xvmt12+MMR5JH/gvKC9CBBvnN8YYr6QP/AXZGSwqzbfSDcYY45X0gR88N3LtbLSpGI0xBlIk8K+qLKKle5CmdpuK0RhjUiLw+yp12ji/McakSOBfOqeATEeajfMbYwwpEviz0h0sm1topRuMMYYUCfzgqdvzRlMHI6N2gdcYk9pSKPA76Rkcod5tUzEaY1Jb6gT+Ss8FXhvuMcakupQJ/AtL8ijISrcLvMaYlJcygT8tTbigoshSOo0xKS9lAj94hnv2Hu+kf8imYjTGpK7UCvwVToZHlTePd8a6KcYYEzMhBX4RuUpE9onIQRG5N8j6pSLyiogMiMg9Aes+LyJ7ROQNEfmliGSHq/GTtcp7gXeXjfMbY1LYhIFfRBzAfcDVwHLgBhFZHrBZK3An8J2Afcu9y2tVdQXgwDPhekzMLsqmrCCLnTbOb4xJYaH0+C8CDqpqg6oOAk8A1/lvoKrNqroVGAqyfzqQIyLpQC5wbJptnpaaSqdl9hhjUloogb8ccPk9b/Qum5CqNuH5FnAUOA50qOp/BNtWRG4VkW0iss3tdofy8lOyqtJJQ0sPHX3BPqOMMSb5RfTirogU4/l2sACYC+SJyI3BtlXVB1S1VlVrS0tLI9amlRWeqRh323CPMSZFhRL4m4BKv+cV3mWhWA8cUlW3qg4BTwMXT66J4bWy3HOB1+bgNcakqlAC/1agWkQWiEgmnouzz4b4+keBtSKSKyICXAnsnVpTw6MoN4OFJXlWusEYk7LSJ9pAVYdF5A7gOTxZOY+o6h4R2eBdv1FEZgPbgEJgVETuBpar6hYReQrYDgwDO4AHIvS7hKym0slLB1ti3QxjjImJCQM/gKpuAjYFLNvo9/gEniGgYPt+A/jGNNoYdisrinhmRxMnOvqZXRSz2wqMMSYmUurOXR+r1GmMSWUpGfiXzykkPU3sAq8xJiWlZODPznCwdE4BuyzwG2NSUEoGfvAUbNvl6mDUpmI0xqSY1A38lU66BoZpaOmJdVOMMSaqUjbw+yp1Wt0eY0yqSdnAv6g0n9xMh43zG2NSTsoGfkeacEF5EXVWs8cYk2JSNvCDZ7hn77FOBoZtKkZjTOpI6cC/ssLJ4Mgobx3vinVTjDEmalI68NdUeko02zi/MSaVpHTgL3fmUJKfSZ3LxvmNMakjpQO/iFBT4bTSDcaYlJLSgR884/z17m66+m0qRmNMakj5wF9TWYQq7G6y4R5jTGqwwF/hu4PXAr8xJjWkfOAvzstk3sxcK91gjEkZKR/4wTPObymdxphUEVLgF5GrRGSfiBwUkXuDrF8qIq+IyICI3BOwzikiT4nIWyKyV0TeEa7Gh0tNRRHHOvpp7uyPdVOMMSbiJgz8IuIA7gOuBpYDN4jI8oDNWoE7ge8EeYl/A/5dVZcCNcDeabU4AsYqdVrdHhNDJzv7OWmdDxMFofT4LwIOqmqDqg4CTwDX+W+gqs2quhU4IydSRIqAy4CHvdsNqmrcjamcP7cIR5rYcI+JqS8+uZO7n6iLdTNMCggl8JcDLr/njd5loVgAuIGfiMgOEXlIRPKCbSgit4rINhHZ5na7Q3z58MjJdLBkVoFNvm5i6mBzN/tPWt0oE3mRvribDqwB7lfV1UAPcNY1AgBVfUBVa1W1trS0NMLNOtuqyiJ2utpRtakYTfT1D41worOfUz2DdNrNhCbCQgn8TUCl3/MK77JQNAKNqrrF+/wpPB8Ecaemwkln/zCHT/XGuikmBTW19409PtJi70ETWaEE/q1AtYgsEJFM4Hrg2VBeXFVPAC4ROc+76ErgzSm1NMJWem/ksnF+Ewuu1tPB/vApmwfaRFb6RBuo6rCI3AE8BziAR1R1j4hs8K7fKCKzgW1AITAqIncDy1W1E/gc8Jj3Q6MB+GSEfpdpWTIrn7xMB7/Z3sR7V84lLU1i3SSTQlxtp3v8h1ss8JvImjDwA6jqJmBTwLKNfo9P4BkCCrZvHVA7jTZGRbojjb+/eilf/90efvDfB7lrfXWsm2RSSGNrL5npaThzMjhkPX4TYXbnrp+b1s7j71aX870/7+cvbzXHujkmhbjaeqkozmFBSR5H7DpTTPx570keeKE+1s2ICgv8fkSEb33gApbNLuSuJ3ZwxHpeJkpcrX1UFueyoCTPhnpi5GevHOE7/7GfoZHRWDcl4izwB8jJdPDjmy5ERPjMz1+nb9AmYjeR52rrpXJGDvNm5llKZ4zUu7sZHE6NObgt8AdROSOXf7t+FftOdvHlp3dZbr+JqK7+Idp7h7w9/lzAUjqjrX9oZCylti4FMvss8I/jneeV8YX1S/ht3TEefflwrJtjkpir1RNwKmfkMm+m58Z2S+mMrkMtPfj6d3VHLfCntNvftZj1y8r4pz/uZevh1lg3xyQpV5und19ZnMt8X+C3cf6oqnd3A1DuzEmJObgt8J9DWprw3Y+sonJGLrc9tt3KNpuI8N28VTkjh5xMB7MLsy2lM8oa3D2IwHWr5lLv7k76aywW+CdQmJ3BxhsvpLt/mNse287gcPJf8TfR1djWR0FWOkU5GQDMm5lrKZ1RVu/uZm5RDmsXzvTMwZ3kJdot8IfgvNkFfPtDK9l2pI3/tSnuphMwCe5oay8VM3IR8dwtbimd0dfg7mFRWf7YHNzJXqnXAn+I3lszl1suXcBPXz7MMzsaY90ck0Rcrb1UFueMPbeUzuhSVRrc3SwsyaMoN4OFJXkW+M1p9169lLcvmMGXn97Nm8c6Y90ckwRUlca2Pipn5I4ts5TO6DrZOUDP4AiLyvIBqKl0UpfkJdot8E9CuiONH350Dc6cTD7zi2209w7GukkmwbV0D9I3NEKVX+C3lM7o8mX0LCr1nPdVlU7cXQMc70jeZA4L/JNUWpDFj25cw4mOfu7+VR2jo8nbKzCRN5bKOeP0UI+ldEbX6cB/uscPsDOJh3ss8E/BmqpivvHe83l+n5vv/flArJtjEthYKmfx6R6/L6XTJgWKjgZ3D/lZ6ZQVZAGwbE4BmY60pB7nt8A/RR97exUfvrCC7//5AH/eezLWzTEJqtFbh7/CL/CDJ6XThnqio97dzcLSvLGsqqx0B8vmFlrgN2cTEf7x/StYUV7I3b+qs6/lZkpcrb2U5GeRk+k4Y7mldEZPg7tnbJjHZ1VFEbubOhhJ0qFcC/zTkJ3h4P6PXYgjzVPJs3dwONZNMgnGV5UzkKV0Rkfv4DBN7X0sLMk7Y/mqKie9gyMcaE7OSp0hBX4RuUpE9onIQRG5N8j6pSLyiogMiMg9QdY7RGSHiPwhHI2OJ5UzcvnBDas50NzFvb/ZndQpYCb8fHX4A1lKZ3Qc8n6r8qVy+ozdyJWkBdsmDPwi4gDuA64GlgM3iMjygM1agTuB74zzMncBSXvL67rqUr74nvN4ducxntjqinVzTIIYGVWOtfeN2+MHS+mMtHq3N/AHDPUsKMmjMDs9aQu2hdLjvwg4qKoNqjoIPAFc57+Bqjar6lbgrO+lIlIB/C3wUBjaG7due+cils0p5JntTbFuikkQxzv6GB7VoD1+S+mMjvrmbkQ8F9P9iYj3Rq7krNkTSuAvB/y7sY3eZaH6HvAlIKmrm4kI7zqvlO1H2+gesLF+MzH/OvyBLKUzOhpaeqgsziU7w3HWutWVTvad6EzKa3cRvbgrItcCzar6egjb3ioi20Rkm9vtjmSzIubS6hKGR5VX60/FuikmAfjX4Q/GUjojr77Zk8oZTE2lk1GFN5qSrzxLKIG/Caj0e17hXRaKS4D3ichhPENEV4jIL4JtqKoPqGqtqtaWlpaG+PLx5cJ5xeRkOHjxQGJ+cJnocrX2kiYwx5kddL2ldEbW6KhyqOXsVE4f3x28da62aDYrKkIJ/FuBahFZICKZwPXAs6G8uKp+WVUrVHW+d7//VtUbp9zaOJeV7mDtwhm8eKAl1k0xCcDV2sucohwyHMH/DC2lM7KOd/bTNzQybo+/JD+LiuIcdibhOP+EgV9Vh4E7gOfwZOY8qap7RGSDiGwAEJHZItIIfAH4mog0ikhhJBsery6tLqWhpYfGNhubNefmague0eNjKZ2RVd98Zo2eYHyVOpNNSGP8qrpJVZeo6iJV/ZZ32UZV3eh9fMLbsy9UVaf3cWfAazyvqteG/1eIL5dVlwCw2Xr9ZgKu1t4zqnIGspTOyGpwTxz4V1c6aWrvo7kruSp12p27Yba4LJ/Zhdk23GPOqX9ohOaugXEv7IKldEZavbuHgux0SvIzx93mdKXO5BruscAfZiLCpdUlvFTfkrR1Psz0+YqzBUvl9LGUzshqaOlmUWn+WHG2YFbMLcKRJklXotkCfwSsqy6hvXeIN5qSq5dgwidYHf5gLKUzcuqbe8a9sOuTk+ngvFkFSXcHrwX+CLhksXec/6AN95jgGoPU4Q9mQUkeRyzwh133wDAnOvvPOb7v47vAm0yTLlngj4CS/CzOn1vIC/stn98E52rrIys9jVLv5B/jmTczj5buQbospTOsDo1ToyeY1ZVOuvqHOZREH8AW+CNkXbWnfEOPlW8wQbhae6kozjnn+DKcTuk8bCmdYRU4z+65jN3IlUSVOi3wR8i66hKGRpQth6x8gzmbpw7/uYd5wFI6I6XB3Y0jTaiaOfH/weKyfPIyHUk1zm+BP0IunFdMdkYaL+y3cX5ztvHq8AeylM7IqHf3UFmcQ1b62cXZAjnShAsqipLqRi4L/BGSneHgogUz7QKvOUtH3xAdfUMTZvSApXRGSr27O6TxfZ9VlcXsPd5J/9BIBFsVPRb4I+iy6hIONndzrL0v1k0xccQVYkaPj6V0hpevONtEqZz+VlUWMTSivHk8OSp1WuCPoHXVniqjVr7B+Gscy+EPLfBbSmd4NbX3MTA8OukeP5A0N3JZ4I+gJbPyKSvI4kUb7jF+xiZgCbnHbymd4TSW0VMWeuCfXZTNrMKspBnnt8AfQb7yDZsPuJPq5g8zPa62Xgqy0ynKzQhpe0vpDC/fPLsLS0If6gHPBOzW4zchWVddQlvvEHuOJcfYoJm+iapyBrKUzvBqcHfjzM1gRt74xdmCWVXl5PCpXtp7ByPUsuixwB9hvvINLx60u3iNh6sttFROH0vpDK96dzcLS/ImvHku0KoK34xcid/rt8AfYWUF2SybU8iLls9vAFWlsa03pFROH0vpDK8G9/jTLZ7LBRVFiCRHiWYL/FGwrrqE14+00Tto5RtSnbt7gP6h0ZAzenwspTM8OvuHaO4aYOEUAn9BdgaLS/OTYg5eC/xRsK66hMGRUbYcao11U8w59A4OR/wGnclm9PhYSmd4NIwVZ5vchV2fVZVOdjZ2oJrYyRohBX4RuUpE9onIQRG5N8j6pSLyiogMiMg9fssrReQvIvKmiOwRkbvC2fhE8bb5M8hMT7Phnjh3y6Pb+Pyv6iJ6jMYQ6/AHspTO8GiYQiqnv5pKJ609g2Mf4IlqwsAvIg7gPuBqYDlwg4gsD9isFbgT+E7A8mHgi6q6HFgL3B5k36SXneHg7QtmsNku8Mat/qERth5u5aWDLRFNvfXdtVsx6R6/d+J1G+eflnp3N+lpMqmsKn+rfJU6E7xgWyg9/ouAg6raoKqDwBPAdf4bqGqzqm4FhgKWH1fV7d7HXcBeoDwsLU8w66pL2H+ymxMdyTVpc7J483gnQyNKZ/8wDRHMnnG19lFakEV2xsTFwfz5UjoPWWbPtDS4e6iamUuGY2qj3OfNLiArPS3hSzSH8tuXAy6/541MIXiLyHxgNbBlnPW3isg2Ednmdidfz/jSxd7yDXYXb1zy/0PefjRyF+9cbb1UFk9umAcspTNcPKmcUxvmAchwpLGivCjhSzRH5eKuiOQDvwHuVtWgdzKp6gOqWquqtaWlpdFoVlQtnV1ASX4WLx5Ivg+1ZLCzsZ2ygiwKstPZEcHeXKh1+ANZSuf0jYwqh1t6WVQ2tQu7PqsqnbzR1MHQyGiYWhZ9oQT+JqDS73mFd1lIRCQDT9B/TFWfnlzzkkdamnDp4plsPhDZMWQzNXWudlZXOVlV6WRHhHr8wyOjHGvvn3RGj4+ldE5PY1svgyOjLJpGjx88F3gHhkfZd6IrTC2LvlAC/1agWkQWiEgmcD3wbCgvLp5b4x4G9qrqd6fezOSwrrqUUz2D7D1h5RviSWvPIEdO9bKqspg1VcXsP9lFdwSmzDze0c/IqE46o8fHUjqn53Rxtun1+FdXJv4dvBMGflUdBu4AnsNzcfZJVd0jIhtEZAOAiMwWkUbgC8DXRKRRRAqBS4CbgCtEpM77c03Efps4t67aW77ByjTHFV/hrVWVTlZXORlV2BWBP+rJ1uEPZCmd09MwVpxtej3+iuIcZuRlJnTgTw9lI1XdBGwKWLbR7/EJPENAgTYDkyuIkcTKCrM5b1YBLx5ws+HyRbFujvGqc7Uj4rklf2TEMwy3w9XOxd46S+HimmQd/kD+KZ0ryovC1q5UUe/uZkZeJsWTLM4WSEQ8N3IlcOC3O3ejbF11CVsPt9E3mBxTuCWDOlc7S8oKyM/ylEpeVJoXkXF+V2sfjjRhTlH2lPa3lM7pqXf3TPmO3UA1FU4OursT9tuXBf4oW7eklMHhUV47bOUb4oGqsrOxfezGHIDVVcVsP9oe9tvyXW29zHVmkz7FHPJ5M311+S3wT0XDNFM5/dVUFqEKuxsTs2CbBf4ou2j+DDIdaWy2tM644KmvPsSqKv/A77kt/2hreFMnXa29Ux7fB8jNTGdWYZaldE5BR+8QLd2D076w6+PrKOxI0OEeC/xRlpPp4G0Liu0Cb5zwjdPWVPgFfu/8quHO559sHf5g5s/Ms5TOKahv8WT0hKvH78zNZP7M3IQd57fAHwOXLi7lrRNdNHda+YZYq3O1k5PhYMms0wHhvNkF5GY6wjrO3z80grtrYMqpnD6W0jk19c3TK84WjKdSpwV+EyJfWqeVb4i9Ha52LqgoOmPc3ZEm1FQ42R7GHn/jNDN6fCylc2oaWnrIcMiUymWMp6bSycnOAY53JF6lTgv8MbB8TiEz8zJtuCfGBoZH2Husc+yGHH+rq5zsPd4ZtuwrXxnfyVblDGRVOqemvrmbeTPzpnxhPRjfOH8iDvdY4I+BtDThksUlvHigJeEndEhke493MTgySk2QwL+mqpjhUeWNY+HJ2nBNsQ5/IEvpnJqGlvClcvosm1NIhkMS8gKvBf4YWVddQkv3AG8lcL2PRFfnHcNfFSTw+7J8wjXO72rtJTsjjdL8rGm9ji+l08b5Qzc0MsqRUz1Tmm7xXLIzHCybU2g9fhO6ddWeCqRWrTN26lyeipzBbqgqyc+iakYu24+E54/6aGsvFcW5eMpXTZ0vpfNQiw31hMrV2svQiE5pgvWJrKp0sruxg5EEK7xogT9GZhdlU12Wb+P8MVTn8ty4NV4wXl3lZPvRtrAMx7la+8J2YdFSOidnrEZPmId6wJMG3DM4wkFv1lCisMAfQ+uqS3ntUGvEJ/g2Z2vvHeTwqd6g4/s+a6qKae4a4HgYZk2bah3+YCylc3LGqnKGKYffn29IsM4Vucl7IsECfwytqy5hYHiUrVa+Iep8lRWDZfT4rPb+UU93Rq6O3iG6+oenffOWj6V0Tk6Du4eS/CyKcjPC/toLZuZRkJ1OnSuxSjdY4I+hty+cQYZD2GzDPVHnX5FzPEtnF5KVnjbtO3jDldHjYymdk1Pv7o7IMA94MvRWVToTrkSzBf4Yys1M58J5xbxggT/qdrraqS7LpyB7/F5gZnoaKyuKpp3ZM1aHP0xDPZbSOTmeVM7wD/P41FQ42X+yi97B8E/eEykW+GNsXXUpe4934u4aiHVTUoaqUudqP6M+z3hWVxXzxrFOBoanfh1munX4A1lKZ+haewZp7RkMew6/v1WVTkZGlT3HEmdmPQv8MXaZN63zJSvfEDVHW3tpC6jIOZ7VlU4Gh0d5cxp/1K7WPopyMig8x7eLybCUztA1+C7sRrDHv7LSM1xYF+aifpFkgT/Gzp9bSHFuBi9YPn/U1PlNtTiR1VXTr9TpyegJX40YsJTOUPlSOSMZ+MsKsil35lCXQAXbQgr8InKViOwTkYMicm+Q9UtF5BURGRCReyazb6rzlW/YbOUboqbO1U52RhrnzSqYcNvZRdnMLcqe1m35063DH4yldIam3t1NZnoa5WEszhbMqkpncvX4RcQB3AdcDSwHbhCR5QGbtQJ3At+Zwr4pb111Cc1dA+w/mVg3gSSqOlc7F5QXhVywa3VVMduPTO0Cr6rS2NYXtvF9H0vpDE29u4cFM/NwpEV26u+ayiKa2vsS5lpdKO/8i4CDqtqgqoPAE8B1/huoarOqbgUC34UT7mvgUivfEDUXhw7fAAAZjUlEQVSDw6PsOdYZ0jCPz+oqJ03tfVOaP8HdNcDA8GhYywGDpXSGqiGCqZz+Vnkn70mUuj2hBP5ywOX3vNG7LBQh7ysit4rINhHZ5nanVgAsd+awqDTPyjdEwd7jnQwOj479oYZibJx/Cn/Uvoyeigj0+MFSOs9lcHiUI629ER3f91lRXogjTRJmYpa4ubirqg+oaq2q1paWlsa6OVG3rrqULYdOWfmGCPP9YYaS0eNz/lxv+d0pjOH66vCHe4zfUjondrS1l5FRjUqPPzcznSWzChLmRq5QAn8TUOn3vMK7LBTT2TelrKsuoX9odMpjySY0dUfbKcnPYm6Qipzjyc5wsHxu0ZRKN/gmbK8I81CPpXROrD4KqZz+3ja/mK2HW2ntGYzK8aYjlMC/FagWkQUikglcDzwb4utPZ9+UsnbhTDIcwiMvHUqoOwATzUQVOcezpsrJrsZ2hkdGJ7Wfq7WXsoIssjMck9ovFPNnWmbPuUSyKmcwN62dR//QKI++fDgqx5uOCQO/qg4DdwDPAXuBJ1V1j4hsEJENACIyW0QagS8AXxORRhEpHG/fSP0yiSwvK50vvPs8/vxWM9d+fzNvNCVW0adE0NE7RENLz1jxtclYXVVM/9DopCfOCWdVzkALSiyX/1zq3d2UFWSdsyxHOFXPKmD9sjIefeVw3HfeQhrjV9VNqrpEVRep6re8yzaq6kbv4xOqWqGqharq9D7uHG9fE9xn37mIx29ZS+/gCB/40Us8+EIDowk2wUM8Gxvfn0RGj4+viudk6/aEsw5/IEvpPLcGd3fUhnl8Nly+iPbeIX611TXxxjEUNxd3jcc7Fs3kT3et413nlfGtTXv5xE9eo7lr+vXgTWgVOcdTUZxDSX7WpC7wDo2Mcrwj/Dn8PpbSOT5Vpd7dE7VhHp/a+TOonVfMQy8eYmiSw4LRZIE/DhXnZfLjmy7kWx9YwdbDrVz9vRf5y1vNsW5WwqtztbOoNH9KNXNEhDVVzkmldB5v72dUw1ecLZCldI7vVM8gHX1DUe/xg6fX39Texx93HY/6sUNlgT9OiQgfe/s8fn/HpZQWZPHJn27lH36/x9I9p8hXkXMqwzw+q6uKOdTSQ1uIWRtjVTnDnMrpYymd44v2hV1/Vywto7osn41/rY/bMiwW+ONc9awCfnv7Jdx88Xx+8tJhPvCjlznYPLkLjAYa2/po7RmcZuD3jvOHOM3e6Tr8kRnjt5TO8UU7ldNfWprwmcsX8daJLp7fH583o1rgTwDZGQ6++b7zeeTmWpo7+7n2B5t5fMvRuO1NxKMdk6jIOZ6VFUU40kK/kcvV1kt6mjCnKHIFwiylM7gGdzdZ6WmUOyNbnG0876uZy5yibDY+Xx+T40/EAn8CuWLpLP501zreNn8GX3lmN5/9xXbae+P/ZpF4UHe0naz0NM6bPXFFzvHkZqazdHZB6IG/tY+5zpyIFgizlM7g6t09LCjJIy3CxdnGk5mexqcuXcCWQ63TnsEtEizwJ5iywmwe/eRFfPWaZfz5rZNc9b0XeaX+1KRfp7N/iO1H23hym4t//tNebnl0K1d853k+8/NtSXkPQZ2rjQvKi8gIsSLneFZXeeZXHQkhzTYSdfgDWUpncA3ubhaVRX+Yx9/1F1VRmJ3Oxr/GX68/PdYNMJOXliZ8+rKFrF04kzuf2MFHH3qV29+5mLvWV58R2FSV5q4BDjZ3j/3Uuz3/NvuVj810pDG/JJdFZfm8Un+K5/ac5N3LZ3HXldWsKJ986mO8GRoZ5Y1jnXx87bxpv9bqymJ+8epRDjZ3T/jtwdXax/plZdM+5rn4p3Qmw/9VOAwMj3C0tZf31cyNaTvys9L5+Dvmc9/zB6mPwT0F52KBP4FdUFHEHz53Kf/w+z388C8Heam+hfcsn+0J8u5uGpq76Ro4fQdhQVY6i8ryuWxJKYvL8llUms/isnwqi3PGatN39g/x05cO89CLDVz7ZnJ8ALx1vIvB4VFqpjG+77Nmnm9GrrZzBv6+wRFaugcilsrp45/Smcj/R+F05FQvo0rMe/wAN18ynwdfbODBFxr43x9cGevmjLHAn+DystL59odquGxJKV9+ejc7jr7FrMIsFpfl84E15Swuy2exN8CXFmRNWKOmMDuDO6+s5uZL5p/xAbB+2SzuXp+YHwB13iyc6VzY9Zk/MxdnbgY7jrZz/UVV4243Vo45wjM/WUrn2Xzz7C4siX3gL8nP4sO1FTy5tZEvvHsJZYWhFweMJAv8SeLalXO5YmkZw6Malkm9/T8AHn3pMA++2MC1P0jMD4AdrnZK8jPDEoRFhNWVzgkrdZ5O5Yxsj99SOs9WH8Mc/mA+vW4hj285ysMvHeLLVy+LdXMAu7ibVHIz08MS9P0VZmfwuSur2XzvFXzx3Ut47dAprv3BZm55NHEuAu+cYkXO8aypKuZAczcdfeNfUB0L/BG6ecufpXSeqd7dzZyibPKy4qNfO29mHldfMIfHXz1KZ5xchLfAb0IS+AGw9XCr9wNgK7sb4/cDoKNviHp3DzUV0x/m8fHNyLXrHLMtudr6yMlwUJKfGbbjjsdSOs8Uixo9E/ns5YvoGhjmsVePxropgAV+M0ljHwB//y7uec8Sth5u470/jN8PgF1TmHFrIisrixDhnPn8rtZeKopzwvYt41wspfM0VaWhOb4yaABWlBdx6eISHnnpEAPDsS+7YoHfTElBdgZ3XHH2B8Cdv9wRVzeV1XmD88ow9vgLszOoLss/5zi/qy1yVTkDWZXO09zdA3QNDLOwJL56/OAp3ubuGuCZ7bGfhNACv5kW/w+AO6+sZtPu41z1vRfZHCcTx+9sbGdRaR5FOeG99rGmqpgdR9uDls1QVRpbe6mKUuC3Kp2n1Td7zkE8pHIGumTxTFaUF/LACw0h3QAYSRb4TVgUZGfwhXcv4be3X0J+djo3Prwl5tVEfRU5w5G/H2h1lZOOvqGgwbajb4iugeGIp3L6WErnaQ0t3lTOOBvqAU9G2IbLF9HQ0sN/vnkipm0JKfCLyFUisk9EDorIvUHWi4h837t+l4is8Vv3eRHZIyJviMgvRSQ+EllNRKwo99xU5qsm+t4fxG4ayca2Plq6B8dmzwon3wXe7UHG+V2tfUDkUzl9LKXztPrmHnIyHMyJk3z5QFedP5uqGbnc/9eGmBZZnDDwi4gDuA+4GlgO3CAiywM2uxqo9v7cCtzv3bccuBOoVdUVgAPPhOsmifmqif7sf1xEZ/8QH/jRS9z3l4NR/3pbN1aRszjsr724NJ+CrPSgBbgiXYc/GEvp9Gho6WZhaeyKs00k3ZHGpy9byE5XO682tMasHaH0+C8CDqpqg6oOAk8A1wVscx3wM/V4FXCKyBzvunQgR0TSgVzgWJjabuLcZUtKee7uy3jP8tn863P7+MiPXxnLb4+Gna52MtPTWDpn6hU5x5OWJqyqcgbN7Il0Hf5gLKXTo97dHZfDPP4+fGEFJfmZ/PiF2BVvCyXwlwP+Mwc3epdNuI2qNgHfAY4Cx4EOVf2PYAcRkVtFZJuIbHO743PyAjN5ztxMfvjR1fzfj9Sw70QXV33vBZ7c5orK19w6Vzsr5hZOuyLneFZXOnnrRCe9g8NnLHe19eLMzaAgzDfTnYuldEL/0AiNbX0sirMc/kDZGQ5uvng+z+9zs/d4Z0zaENGLuyJSjOfbwAJgLpAnIjcG21ZVH1DVWlWtLS0tjWSzTJSJCB9YXcGf7l7HivIivvTULjb84nVOdQ9MvPMUDY2MsrupIyLDPD6rq4oZVdjpOvMahqu1L6rDPGApnQCHT/WgGp8XdgPdtHY+eZkOfhyjks2hBP4moNLveYV3WSjbrAcOqapbVYeAp4GLp95ck8gqinN5/NNr+co1S/nLW27+JoKTyO870cXA8GhYb9wK5Cv6FjgVYzTq8AeylE6/VM447/EDFOVmcMNFVfx+1/GoDn/6hBL4twLVIrJARDLxXJx9NmCbZ4GPe7N71uIZ0jmOZ4hnrYjkiucWxiuBvWFsv0kwjjTh1ssW8bs7LmFmXiaf/OlWvvrM7rOGS6bLd2E3Ehk9PsV5mSwsyTtjnH90VGmMQY/fUjrjqypnKD61bgECPLz5UNSPPWHgV9Vh4A7gOTxB+0lV3SMiG0Rkg3ezTUADcBB4ELjNu+8W4ClgO7Dbe7wHwv1LmMSzbE4hv7vjEj69bgGPv3aUa7+/eSxYh0Odq50ZeeGpyHkungu8bWPXLJq7BhgcGaUiSqmcPqme0tnY1svvdh6jckYOOZmOWDcnJHOKcrhuVTlPbD1Ka09073YPaYxfVTep6hJVXaSq3/Iu26iqG72PVVVv966/QFW3+e37DVVdqqorVPUmVY3cwK5JKNkZDr76t8t57Ja30z80wgfvf5kfPX8wLBd+68JckXM8a6qKaekepLHNk7t/OpUz+pN8V5cV8Nf9bg6c7Ir6sWPp5foW3vfDlzjZ0c8/vf+CWDdnUjZcvpD+oVF+9srhqB7X7tw1MXfxohL+dPdlXLViNt/+93188cmd0ypk1dk/RL27OywTr0xktfcagq9uT7Tq8AfzlWuWIQIf2vgKWw/HLkc8WlSVhzcf4qaHX2NGXia/u+MSLl+SWIkh1bMKWL+sjEdfPhz24c5zscBv4kJRTgY/vGE1X3z3Ep7e0cRHH9xCyxSzfnY3dqAanhm3JnLerAJyMhxj4/y+u3bLndHv8S+fW8jTn72YmXmZ3PjQFv79jdiWBYik/qERvvDkTv7xD29y5dIynrnt4oTI5glmw+WLaOsd4smtrok3DhML/CZuiAifu7Ka+z66hj3HOnj/fS+x78Tkhy181wrCWYN/POmONFZWFI3dwetq62VWYRbZGbEZZ66ckctTn72Y5XML+exjr/PzVw7HpB2R1NTex4c2vswzO5r4/PolbLzxwqjeMxFutfNnUDuvmAdfPMTQyGhUjmmB38Sdv105hyc/8w4Gh0f54P0vTzrlc8fRdhaW5FGUG51gsGZeMXuOddI/NIIrilU5xzMjL5PHb1nLFeeV8T9/t4d/fe6tmNaFCadX6k/x3h9s5khLLw99vJa71lfHbXmGydhw+SKa2vv4467jUTmeBX4Tl1ZWOPndHZcwb2Yun3p0Kw9vPhRS8PJV5IzGMI/P6konw6PKnmMdNLZFP5UzmJxMBz++6UKuf1sl9/2lni89tStqvclIUFV+8tIhbnx4C8W5Gfz2jktYv3xWrJsVNlcsLaO6LJ+Nf62Pyod0fExKaUwQc4py+PWGd/D5X9Xxj394k3p3N//wvvPPWYLhWEc/Ld0DEb1xK5CvUudrh9o43tEX9VTO8aQ70vjnv7uA2UXZfO+/DuDuHuC+j66Jm7loQ9U/NMJXntnN09ubWL9sFv/3IzUJPbQTTFqacNf6at481snA8GjEhwqtx2/iWm5mOvd/7EJue+ciHt9ylJt/8hodvePXo/HNuBWN8X2f0oIsKmfk8MfdxxjV2KRyjkdEuHv9Ev757y7ghf1ubnjw1SlfNI+FY+19fHjjKzy9vYm711fzwE2JPZ5/LteunMuXrloaletDFvhN3EtLE7501VL+z4dreO1QKx/40Uvjliaoc7WRmZ7GsjmFUW3j6spi3mjyFNyKRSrnRG64qIoHbqpl/8kuPnT/ywlxh++rDZ7x/EMtPTz48VruXr8kKcbz44EFfpMwPnhhBY9/ei3tfUO8/76XeLn+7Okdd7o6OH9uIZnp0X1rr/YbWorHwA+wfvksHrtlLR19Q3zw/pfZ3RibCXImoqr89KVD3PjQFopyM/jt7Zfw7iQaz48HFvhNQnnb/Bn89rZLKCvI4uMPv8YvXzs6tm54rCJn9IZ5fHzj/BkOYXaczv4EcOG8Yp767MVkpTv4yAOv8Nf94SuBfqp7gIPNXZzs7KdnYHhKFyn7h0a459e7+Obv3+TyJaX89vZLWByH8+cmusS6ymMMUDUzl9/cdjGfe3wHX356Nwebu/nKNcvYd7KLvqGRmAT+5XM83zLmFGXjiPPhiEWl+Txz28V84idb+dRPt/IvH1zJBy+smNRrdA8M80ZTBztd7exq7KDO1U5Te98Z26QJ5GelU5CdQUF2OgXZ6Wc8z89OpzA7w7ssndxMB/c/X8/Oxg7uvLKau69MjlTNeGSB3ySkwuwMHv5ELf/0x708vPkQh1p6WLtwBhCdO3YDZaan8fYFM8hPkIyZssJsnvzMWjb84nW++OudnOzq57OXLwpa22hweJS3TnSys9EX6Ns50NyNr0NfUZzDqionn7h4HrMKs+keGKarf5ju/mG6+j0Tz/ueu7sHaGjp8a4bZjAgxTTPm4b6N+fPjsZpSFkSjzd21NbW6rZt2ybe0BjgF68e4RvP7mFUleLcTF7/2vqIF2cLpndwGEESpjokeIL6Pb/eybM7j/GJd8zjf167nMOnescCfF1jB3uPdY4F6Jl5maysKKKm0klNhZOVFUXMzM+a8vEHhkf8PiSGmV2UTWnB1F8vlYnI66paG8q2idE9MeYcblw7j/kz87jtsdd5x8KZMQn64Ek9TTSZ6Wl87yOrmFWYxYMvHuKXW10MDnuCfG6mgwvKi7j5kvljQb6iOCes5zcr3UFWvoOSaXx4mMmzHr9JGp39Q6SJJMxwS7z59TYXu5s6WFFexKpKJ4tK8+P+eoU5zXr8JiUVJumNPdHy4dpKPlxbOfGGJuFZOqcxxqQYC/zGGJNiQgr8InKViOwTkYMicm+Q9SIi3/eu3yUia/zWOUXkKRF5S0T2isg7wvkLGGOMmZwJA7+IOID7gKuB5cANIrI8YLOrgWrvz63A/X7r/g34d1VdCtTgmbDdGGNMjITS478IOKiqDao6CDwBXBewzXXAz7yTrr8KOEVkjogUAZcBDwOo6qCqtoex/cYYYyYplMBfDvhPBtnoXRbKNgsAN/ATEdkhIg+JSF6wg4jIrSKyTUS2ud3hqx9ijDHmTJG+uJsOrAHuV9XVQA9w1jUCAFV9QFVrVbW2tLQ0ws0yxpjUFUrgbwL8k3srvMtC2aYRaFTVLd7lT+H5IDDGGBMjodzAtRWoFpEFeIL59cBHA7Z5FrhDRJ4A3g50qOpxABFxich5qroPuBJ4c6IDvv766y0icmQSv4e/EuDsQu3xw9o3Pda+6bH2TU88t29eqBtOGPhVdVhE7gCeAxzAI6q6R0Q2eNdvBDYB1wAHgV7gk34v8TngMRHJBBoC1o13zCmP9YjItlBvW44Fa9/0WPumx9o3PfHevlCFVLJBVTfhCe7+yzb6PVbg9nH2rQMS/kQZY0yysDt3jTEmxSRj4H8g1g2YgLVveqx902Ptm554b19I4rIsszHGmMhJxh6/McaYc7DAb4wxKSYhA/90qoVGqX2VIvIXEXlTRPaIyF1BtnmniHSISJ335+tRbuNhEdntPfZZ053F8hyKyHl+56VORDpF5O6AbaJ6/kTkERFpFpE3/JbNEJH/FJED3n+Lx9n3nO/XCLbvX71VcXeJyDMiEnQW+oneCxFs3zdFpMnv//CacfaN1fn7lV/bDotI3Tj7Rvz8hZ2qJtQPnnsJ6oGFQCawE1gesM01wJ8AAdYCW6LcxjnAGu/jAmB/kDa+E/hDDM/jYaDkHOtjeg4D/r9PAPNief7wFBtcA7zht+zbwL3ex/cC/zJO+8/5fo1g+94DpHsf/0uw9oXyXohg+74J3BPC/39Mzl/A+v8DfD1W5y/cP4nY459ytdBoNVBVj6vqdu/jLjylqAML28W7mJ5DP1cC9ao61Tu5w0JVXwBaAxZfBzzqffwo8P4gu4byfo1I+1T1P1R12Pv0VTylVGJinPMXipidPx8REeD/A34Z7uPGSiIG/ulUC406EZkPrAa2BFl9sfdr+J9E5PyoNgwU+C8ReV1Ebg2yPl7O4fWM/wcXy/MHMEu9pUnwfCuZFWSbeDmP/wPPN7hgJnovRNLnvP+Hj4wzVBYP528dcFJVD4yzPpbnb0oSMfAnDBHJB34D3K2qnQGrtwNVqroS+AHw2yg371JVXYVnEp3bReSyKB9/Qt4yH+8Dfh1kdazP3xnU850/LnOjReSrwDDw2DibxOq9cD+eIZxVwHE8wynx6AbO3duP+7+lQIkY+KdTLTRqRCQDT9B/TFWfDlyvqp2q2u19vAnIEJGSaLVPVZu8/zYDz+D5Su0v5ucQzx/SdlU9Gbgi1ufP66Rv+Mv7b3OQbWJ6HkXkZuBa4GPeD6ezhPBeiAhVPamqI6o6Cjw4znFjff7Sgb8DfjXeNrE6f9ORiIF/rFqot0d4PZ7qoP6eBT7uzUxZi1+10Gjwjgk+DOxV1e+Os81s73aIyEV4/i9ORal9eSJS4HuM5yLgGwGbxfQceo3b04rl+fPzLPAJ7+NPAL8Lsk0o79eIEJGrgC8B71PV3nG2CeW9EKn2+V8z+sA4x43Z+fNaD7ylqo3BVsby/E1LrK8uT+UHT8bJfjxX+7/qXbYB2OB9LHjmCa4HdgO1UW7fpXi+9u8C6rw/1wS08Q5gD54shVeBi6PYvoXe4+70tiEez2EenkBe5LcsZucPzwfQcWAIzzjzp4CZwJ+BA8B/ATO8284FNp3r/Rql9h3EMz7uew9uDGzfeO+FKLXv59731i48wXxOPJ0/7/Kf+t5zfttG/fyF+8dKNhhjTIpJxKEeY4wx02CB3xhjUowFfmOMSTEW+I0xJsVY4DfGmBRjgd8YY1KMBX5jjEkx/w+MPsT8gwlEaAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113561ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(len(eval_losses)), eval_losses)\n",
    "plt.title('test loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1137a9828>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEICAYAAABYoZ8gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4XPV56PHvq91arGW02LJsyZLlDWM2YSwgDoSEGJJCIG0C2WkSSm+gyb1PklLam7T33rQkadOQloSSlISskBBMnIRAIJjN2ODdlo1lLZZtSbZHm7VZ+7z3jzkjBlmSR9Is0sz7eR491sw5M+c349F7fvOe97xHVBVjjDGxIy7SAzDGGBNeFviNMSbGWOA3xpgYY4HfGGNijAV+Y4yJMRb4jTEmxljgN8aYGGOB30QdEWkQkXcH4Xk+JSKvBmNMxswmFviNMSbGWOA3UUVEfgIsAX4rIj0i8mXn/vUi8pqInBGRfSJyjd9jPiUi9SLSLSJHReSjIrIKeAiodJ7nzATbu0NE3nQeWy8ifzVm+c0isldEukSkTkQ2OvfniMgPRaRZRDpE5KkQvSXGnEOsZYOJNiLSAHxGVZ93bi8C9gMfB54BrgMeA1YCZ4GTwOWqWi0iC4EcVT0oIp9ynufqSbb1PuAwUA9sAP4AXK2qu0VkHfAc8OfAn4CFQIaqHhaR3wM9wF3Ov1eq6kvBfSeMGV9CpAdgTBh8DHhaVZ92bj8nIjuBG4EnAA+wRkSOq+pJvDuCgKjq7/1uviQifwTeAewGPg08oqrPOcubAJydyw2AS1U7fI+d3kszZuos1WNiQTHwF06a54yTtrkaWKiqvcCH8c68T4rI70VkZaBPLCI3iMh2EWl3nvdGINdZvBioG+dhi4F2v6BvTFhZ4DfRaGz+8gTwE1XN8vtJU9X7AVT1WVV9D95UzGHg+xM8z9uISDLwa+BfgQJVzQKeBsRvu2XjPPQEkCMiWdN4bcbMmAV+E41OA6V+t38K/JmIvFdE4kUkRUSuEZEiESlwDsCmAQN48+0ev+cpEpGkCbaTBCQDLcCwiNwAXO+3/L+BO0TkOhGJE5FFIrLSSSf9AfiuiGSLSKKIbAjaqzfmPCzwm2j0L8A/OGmdL6rqCeBm4D68QfoE8CW8n/844H8BzUA78E7gr53neQE4CJwSkdaxG1HVbuBvgF8CHcBHgM1+y98A7gD+HejEm8cvdhZ/HBjC+w3DDXwhSK/dmPOyqh5jjIkxNuM3xpgYY4HfGGNijAV+Y4yJMRb4jTEmxszKM3dzc3O1pKQk0sMwxpg5Y9euXa2qmhfIurMy8JeUlLBz585ID8MYY+YMETkW6LqW6jHGmBhjgd8YY2KMBX5jjIkxFviNMSbGWOA3xpgYY4HfGGNijAV+Y4yJMRb4jTFmFnittpUfbj2KxxP6jsmz8gQuY0z0GhrxcKqzn+YzfZzs7KfpTN/o74nxwvpSF5VlLpbnZxAXJ+d/wijQMzDMl57YT1JCHLevW0JKXHxIt2eB3xgTNKpKW+8gJ894A/rJTm9Qbz7TT7Pzu7t7gLGXAclOTaQwax5d/UM8e/A0ADlpSawvzaGyLJfKUhdleWmIROeO4Ot/OExzZx+/+qtKUhJDG/TBAr8xZpraegZ482Q3h052ev9t7qKhrZeBYc/b1ktOiGNR1jwKs+axoTyPwqx5LMqax8KsFAqz5lGYOY95SW8Fu8aOs2yra2NbfRvb69p4+sApAPIzkllf6uLKMu83giU5qVGxI3itrpWfbD/Gp69eSkVJTli2OSuvwFVRUaHWq8eY2WHEoxxt7eXNk10cOtnl/be5C3f3wOg6C+ansGphBsvy052gPm802GenJk47QKsqx9rOsq2+bXRn0OJstzAzhfVlLq4sy6WyzMWirHlBeb3h1DswzHu//TIJccIfPr/hbTvAqRKRXapaEci6NuM3xozqGRjmsC+4n+zi0Mluqk910T/kncUnxAnL8tO5elkuqwvns2qh9ycnbaLr0c+MiFCSm0ZJbhq3r1uCqlLX0su2ula21bfxYnULT+5uAmBJTiqffcdSPl5ZEpKxhMLXnzlM05k+Hr+zckZBf6oCCvwishF4AIgHfqCq949Zng08ApQB/cBfqmqVs+x/Ap8BFDgA3KGq/UF7BcaYSakqXX3DtPT009I9SEvPAC3dA7Q6//p+WnsG3jaLz0pNZNWC+XxkXbET5L0z+uSE8AWosUS8O55l+el8vLIEj0c54u5mW10bT+xq5GtPv8mHLl8c0TEGaltdGz/edow7riph3dLwpHh8zhv4RSQeeBB4D9AI7BCRzap6yG+1+4C9qnqLiKx01r9ORBYBfwOsVtU+EfklcBvwoyC/DmNQVVSJmUqQsQ40dvLHQ6fGCeiDDI54zlk/IU7Iy0gmNz2ZBZkpXLgok6LseaMz+YWZKbM+hx4XJ6xcMJ+VC+azMHMed/10F/sbO7k8TLny6To7OMyXf72PYlcqX3rvirBvP5AZ/zqgVlXrAUTkMeBmwD/wrwbuB1DVwyJSIiIFftuYJyJDQCrQHKzBG+NzqLmLv/7ZLq4sc/Evt66N9HAi4h9/e5DdxztwpSU7AT2Jsvx08jKSyUtPftu/uenJZM5LjKqd5PrSHES8M+nZHvi/8Uw1J9r7ePzO9aQmhT/jHsgWFwEn/G43AleMWWcfcCvwioisA4qBIlXdJSL/ChwH+oA/quofx9uIiNwJ3AmwZMmSKb0IE9t+v/8kX/zVPvqGRujsG+JrH7gwqgJaIFSVI6e7+ci6JXztlgsjPZyIyEpNYtWC+Wyra+NvriuP9HAmtL2+jR+91sCnrizhilJXRMYQrDN37weyRGQvcA+wBxhxcv83A0uBQiBNRD423hOo6sOqWqGqFXl5AV09zMQ4j0f55rOH+dzPd7O6cD5/d8NKzpwd4s1TXREZz9CIh+FxUirh0NI9QHf/MMvy0yOy/dlifamL3cc76B8aifRQxnV2cJi//fV+luSk8uWN4U/x+AQS+JuAxX63i5z7Rqlql6reoaoXA58A8oB64N3AUVVtUdUh4EngyqCM3MS0rv4hPvvjnTy4pY7bLl/Mzz97BTddXAh4v+pHwl/9ZBdfeHxvRLZd6+4BiPnAX1nmYmDYw94TZyI9lHF989lqjrWd5Rt/vjYiKR6fQAL/DqBcRJaKSBLeg7Ob/VcQkSxnGXgreF5W1S68KZ71IpIq3qNE1wFvBm/4JhbVtfTwgQe38tKRFv7vzRfwL7deSHJCPAsz51HiSmV7ffgDf+/AMC8faWHXsY6wbxugtsUb+MvzMyKy/dli3dK38vyzzRtH2/nRaw18srKY9RFK8ficd5ejqsMicjfwLN5yzkdU9aCI3OUsfwhYBTwqIgocBD7tLHtdRJ4AdgPDeFNAD4fklZiYsOWwm7/5xR4SE+L46WeuOOcPqLLMxe/2n2TEo8SHMc//+tE2hj3Kyc5+uvqHmJ+SGLZtg3fGn56cQMH85LBud7bJnJfIBYXzI7Lzn0zf4AhffmIfRdnz+PLGlZEeTmB1/Kr6NPD0mPse8vt9G7B8gsd+FfjqDMZoDKrK916q45vPVrNqwXwe/sRlFGWnnrPe+lIXv3jjBAebO1lblBW28b1a81agqXX3cOmS7LBt27fNsvz0WV9+GQ6VpS4efe0Y/UMjYel7E4hvPltNQ9tZfv7ZK0hLjvx5s9aW2cx6fYMj3POLPXzjmWred+FCfv3XV44b9MH7Rw/h/6q/tbaVYpd3TDWnu8O6bfAG/mV5sZ3f96ksczE44mF3hNJuY+1oaOeHrx3l4+uLubIsN9LDASzwm1museMsH/zea/z+wEm+vHEF/3H7JZOe2p4/P4WyvDS2hfGrvru7n+rT3Xz48sUkJ8RRc7onbNsG74Fud/dAzB/Y9akoySFOmBXpHm+KZz+LsuZx7w2RT/H4WOA3s9b2+jZu+s+tnOg4yyOfvJz/cc2ygFIZlWUudhxtZyhMpZVba1sB2FCex7L8dI64wxv4raLn7eanJHLhosyw7vwn8m9/rOZoay/f+ODaWZHi8bHAb2YdVeXH2xr42A9eJys1kac+dxXXrswP+PGVpbn0Do5woKkzdIP082pNG9mpiaxeOJ/y/HRqw5zqscB/rvVlLvaeOEPfYOTq+Xcda+e/tx7lo1cs4cplsyPF42OB38wqA8Mj3PvrA3zlNwfZsDyPpz53FWVTzF2vL/Werh+OPL+qsrW2lSuX5RIXJ5QXZNDc2U93/1DIt+1T6+4hKT6Oxdlzry1xqKwvdTE0ohErr+0fGuFLv9pPYeY8/u7GVREZw2Qs8JtZw93Vz+0Pb+fxnSf43LVlfP8TFdMqi3SlJ7OiICMsOd66ll5OdfVztTOjW17graOvDWO6p9bdw9LcNBLi7c/Z5/KSHOLjhG31rRHZ/reeO0J9ay9f/+Ba0mdRisdn9o3IxJzBYQ8/e/0YD/yphoEhDw9+5FLet3bhjJ6zsszF4ztOMDjsISkhdAHx1ZoWgNHAX+6kW2pO93BJmEo6a909XLgoMyzbmivSkxO8ef4InMi161gHP3ilntvXLeHq8tmV4vGxKYKJGFXlmapTXP/vL/FPvz3E6oXzeepzV8046IP3q37f0Aj7GkN76v6rtW0Uu1JZnOMt5Vyck+qt7HGHJ8/fPzTCiY6zlt8fR2WZi/2NnfQODIdtm/1DI3zpiX0szJzHfTfOniqesSzwm4jYe+IMH/qvbdz1010kxMfxyKcq+NlnrmDFguC0HPBv0RsqwyMette3cZXfgbv4OKEsL50jYSrprG/pRdUO7I6nstTFsEfZGcY8/78/d4T6ll7u/+CFZIT57O2psMBvwupE+1nu+cUePvDgVo629vK1W9bwzOffwbtWFgT1rFP/Fr2hsq/xDD0Dw6NpHp/lBelhO4nL16PHAv+5LivOJiFOwpbu2XO8g++/Us/t6xbzjvLZ3WHYcvwmLDr7hvjullp+uLUBEbj72mXcdU1ZSA98VZa5+Mn20J26/2pNGyJwZdnb+wWVF2Tw1N5muvuHQj7rq3X3ECewNDctpNuZi9KSE7hocVbYTuT69vM15KYnz8oqnrFsxm9CamjEw4+2HuWab27h4Vfq+bOLCnnxS9fwxfeuCHm1Q2Wpi8FhD7uPh+ar/tbaVi5clElW6tsvNO47wBuOyp46dw+Lc1JnTU+a2aay1MWBpk56Qpznd3f180pNC39RURT2Bn3TYYHfhMRbB25f5h9/e4hVC+fz27uv5t8+dBELM8NTb76u1Dl1PwRf9XsHhtl9vONt+X2fcqeksyYMgd969ExufamLEY+yo6E9pNvZvK8Zj8ItlxSFdDvBYqkeE3R7T5zhn3//Jm80tLMsP51HPlXBtSvyw945MpSn7vvaMI/N7wMsyUklKSEu5Hn+4REPR1t7uWbF7M4nR9Jlxdkkxgvb69q4dkXgZ39P1ZO7m1hblDlnjrVY4DdBoarUtfTynT/VsHlfM7npSXztljV8uGJxRE8sWl/m4pFXj9I3ODJpc7eperWmjeSEOC4rPrdW31fZE+oZ//H2swyOeCibI8EmEuYlxXPJ4uyQ9u2pPtXNoZNdfPXPVodsG8Fmgd9My9CIh4PNXexsaGfXsQ52NHTQ2jNAckIcd1+7jL96Z+msKGerLHXxXy/Vs/NYe1ArLbbWtrJuac6EufXlBensbAhtGaH16AnM+tIc/nNLbcgukLNpTxPxccKfXVQY9OcOFQv8JiCdfUPsPt7BroYOdh5rZ++JM/QPebtfFmXP4x3luVxWnM27VxWwIDMlwqN9y+UlOaMlfcEK/L42zLdcumjCdcrz0/nN3mZ6BoZDdhDbSjkDs77MxXdeqGXH0XauW1UQ1Of2eJTf7G3incvzyE2fO1c/s8BvzqGqNHb0sfNYOzsbOtjZ0MERdzeq3jTG6oXzuX3dEiqKc6goyaZg/uwJ9GOlJSewtii4eX5fG+bx8vs+5X49ey5eHJorgdW6eyiYnzwnqkgi6dIl2SQlxLGtri3ogX97fRsnO/u5bw6UcPqzwG8Ab7+cp/Y28VJ1CzuPtXO6awDw9jy5ZEkW71u7kIribC5anDWr+ooHorLMxUMv1Qdt9u3fhnkivmZtNae7Qxb469w9NtsPQEpiPJcszmL70eDn+Z/c00RGcgLvWR3cHUqoza2/YBN0wyMentzTxHf+VENjRx+FmSlcsdTF5SXZXFacw4oFGWG9aHkoVJbm8uCWOnY0tM+4smNsG+aJjFb2hOgAr+9g+gcnSTeZt1SWuXjgTzV0nh0iMzU435D6Bkf4w4GTvG/twjl3HoUF/hg14lE272vigedraGg7y4WLMvm/N6/hmhV5UXfB7mCW9I1twzyR0cqeEJV0nurqp2dg2Gb8AaosdfHt52t4/Wgb11+wICjP+cdDp+gdHOEDl8y9na8F/hjj8Si/P3CSbz9/hLqWXlYuyODhj1/Ge1YHt1fObBLMkr6xbZgnU56fHrILgfgqeqyUMzAXL8kiOSGO7fXtQQv8m/Y0UZiZwvqlrvOvPMtY4I8RHo/y7MFTfPv5GqpPd1Oen853P3opGy9YMGnKIlqsL3Pxny/UzLikb2wb5sksL0hn875megeGg35cxEo5pyY5IZ7LioNXz9/SPcArNa3cuaF0Tv79WMuGKKeqPHfoNO//j1f565/tZsjj4Tu3X8IzX9jAjRcunJMf2umoLHXhUXijfvqn7o/Xhnkyy/JDdzWuWncP81MSyJtDJYSRVlnq4s2TXXT0Ds74uTbva2bEo9w6B9M8YIE/aqkqW6rd3PzgVj774530Dg7zrQ9dxB+/sIGbLiqc8wdsp+qSJVnekr4ZzPj2NXaO24Z5IssLvLPxIyHI89c4FT3Rmp4LhfVOF9XXj868b8+mPY2sWTR/tGx3rrFUT5TxVp208a3nqtl9/AxF2fP4xgfXcsuli0iM4WuypiTGc9mS7Bn1Zn+1phUR78wxEEtyUkmKjwvJjL/O3cN1q0LXeyYaXVSUxbzEeLbXt7FxzfTz/DWnu6lq6uJ/v3/utGgYywJ/lOjqH2LH0Xb+66V63mhopzAzhX++5UL+/LKikF5zdi6pLHPx788f4czZwXNaKQfC14Y5Oy2wxybEx1Galxb0GX9H7yBtvYOU58/N2WakJCXEUVEys50/eGv34+OEm+ZQi4axLPDPQapK05k+p0eO9+za6tPeM2vzM5L5PzdfwIcvX0xywtyqLQ61yjIX33oOtte3T3nG52vD/NkNpVN6XHlBBnuCfD0Aa9UwfetLXXzz2WraegZwTeP4iMej/GZPE+8ozyUvY+4eXwko8IvIRuABIB74gareP2Z5NvAIUAb0A3+pqlUisgJ43G/VUuArqvrtYAw+VgyPeDh8qpudDe3sOObtl3Oqqx+AtKR4Li3OZuOaBaMtFObaySTh4vuqv62udcqB/42j7RO2YZ7M8vx0fhvkyh6r6Jm+9aVv5flvvHDhlB+//WgbzZ39/O0Ns/dC6oE47ydRROKBB4H3AI3ADhHZrKqH/Fa7D9irqreIyEpn/etUtRq42O95moBNQX4NUadnYJg9x709cnYea2fv8TP0Do4AsDAzhcuX5lBRnM1lxdmsXJAR0bbHc8noV/1pHOB9paZ1wjbMkyl3DvDWtfSwtig4rRtq3T2kJMaxKCs8F7SJJmuLMklNimdbXdu0Av9Te5pIT07g+tXBORcgUgKZgqwDalW1HkBEHgNuBvwD/2rgfgBVPSwiJSJSoKqn/da5DqhT1WPBGXr0+en2Y/z89eMcPtWFR0EEVi6YzwcvK+Ky4mwqSnLsj32GfF/1W3sGptRN8XxtmCfiq/o4cjq4gb80Nz1mSnGDKTE+joqSnGldh7d/aIQ/HDjFxjULgnpth0gIJPAvAk743W4Erhizzj7gVuAVEVkHFANFgH/gvw34xUQbEZE7gTsBlixZEsCwoktn3xBf3XyQ8vx07r52GRUlOVyyJGtW9LSPJpVOSd/2+jbevzawg3OBtGGeSLFT2VPjDt4B3lp3z5S/eZi3VJa6+Pozh2npHphSnv65Q6fpHhies7X7/oKVI7gfyBKRvcA9wB5gxLdQRJKAm4BfTfQEqvqwqlaoakVeXuxdSu6VmhZGPMr/+8Aa/tf1K9iwPM+CfghcuCiTNOerfqBeq/WuO9X8PrxV2VNzOjglnWcHh2k602f5/Rnw3/lPxaY9TSzMTBk9TjCXBRL4m4DFfreLnPtGqWqXqt6hqhcDnwDygHq/VW4Ado9J/Rg/Ww63kDkvMWQtfI1XYnwcly/NmVKe/5Wa1vO2YZ5MeUFG0Gb89S29gB3YnYk1hfNJT06YUuBv7RngpSMt3HRxYVSk2AIJ/DuAchFZ6szcbwM2+68gIlnOMoDPAC+rapffKrczSZon1nk8yktH3GxYnmcHasOgstRFfUsvp53KqMkE2oZ5MuX56Zxo7+Ps4PC0Hu/PtwOxwD99CfFxXD7Fg/y/HW3RUBTCkYXPeaOMqg4DdwPPAm8Cv1TVgyJyl4jc5ay2CqgSkWq8s/vP+x4vIml4K4KeDPbgo0VVcyetPYO8a2XspbgiYSpf9QNtwzwZX+uGYJzBW+vuIT5OKHGlzfi5YlllWeA7f/CmeVYvnM+KBdFx0lxA00tVfVpVl6tqmap+zbnvIVV9yPl9m7N8hareqqodfo/tVVWXqnaG5iXMfVsOtyACG4J4MXAzsQsKM8lISQgozz+VNswT8TVrC0aev9bdQ7Er1c7GniFfnj6QnX+tu4f9jZ3cGkUXvbFPzyzwQrWbi4qypnUmoZm6+DjhiqWugL7qv1rbxpKcwNowT6TElUpivHAkCHn+WncPy/IszTNTvp1/IIF/055G4oQ53aJhLAv8EdbaM8D+xjMzvjKUmZrKMhfH2s7SfKZvwnV8bZivLp/+bB+cyp7cdGpnOOMfGvFwrO2s5feDwLvzzznvtz6PR3lqTzNXl+eRPz8lTKMLPQv8EfbykRZU4V0rLfCHk6/D5mR/+FNtwzyZ8oL0Gc/4j7X1MuxRC/xBsr7URUPbWU52Trzzf6OhnaYzfVFRu+/PAn+EbaluITc9mQsKp1cqaKZn5YIMslMTJ033TLUN82TK8zNo7JhZZY/16AmuQPL8m3Y3kZoUz/UXFIRrWGFhgT+Chkc8vHykhWtW5EVFbfBcEufL808y499a28qawsDbME9meUE6qlDn7p32c4xeZ9dy/EGxeuF8Muclsr1u/Auz9A+N8PSBk2xcs4DUpOhqZGyBP4L2nDhDZ9+Q5fcjpLLMRdOZPk60nz1nma8Nc6CXWTwfX8+emZzIVevuoTAzJejX741VcXHCuklO5nv+TV+Lhuio3fdngT+Cthx2Ex8nMz54aKbHV88/3qzf14b5HUH6vyn2VfbM4ABvbUsPZZbmCarKUhfH28/SNM5B/k27myiYnzz6OYkmFvgjaEt1CxXF2WTOs548kVCen05uetK4M77ptmGeSKKvsmeaM36PR72lnBb4g2r0ZL4xO/82p0XDBy5eFJXXp7bAHyGnOvt582QX11o1T8SICFeUunitrhVVfduyrbWtXF4y9TbMk1lWkD7tGX/TmT76hzx2ucUgW1Ew/kH+3+0/ybBHp9WRdS6wwB8hW6rdAJbfj7DKUhenuwY42vrWQVdfG+Zg5fd9ludncKLjLH2DI+dfeQy73GJoTHSQ/8k9TaxaOJ+VC6Kz2s4Cf4RsOeymMDNltI+LiYzRPL/fjM/XhjlY+X2fcl9lT8vUZ/11VsoZMmMP8te19LDvxBluuSR6ztQdywJ/BAwMj7C1tpVrV+YjEn35w7mkNDeN/Izkt834XqlpJWsGbZgn4tvJT6eyp9bdQ05aEjlBKC01b+er5/ft/J/a00ScwM0XR2eaByzwR8TOhg56B0cszTMLiAiVZS6217ejqqNtmK8qm34b5okUu9KmXdljPXpCZ3lBOq60JLbXteHxKJv2NHHVslwKoqhFw1gW+CNgy2E3SfFxXLks+srE5qLKUhetPQPUuntG2zAHO78P3sqepblp1Jye2oxfVa2UM4REhPWlLrbXt7HzWAeNHX3cEmUtGsayM0Ei4IVqN1eU5kTd2YBzlX+e31fcE+z8vk95fgZVzVPrUN7WO8iZs0OW3w+h9aU5/P7ASR740xHmJcbz3gsWRHpIIWUz/jA71tZLfUuvpXlmkSU5qRRmprCtro1Xalpn3IZ5MuUF6Rxvn1plj/XoCT3fzn9rbRsb1yyI+rOjLfCH2YvV3gt7WDfO2UNEWF/m/ar/en1bSNI8PuX5GVOu7LHAH3pleenkOtfDiPY0D1jgD7st1W6W5qZRkmuXzptNKktddJwdojtIbZgnMp3Knlp3D6lJ8RRmRu/BxkgTEd65PI9FWfNCuuOfLaL7+8ws0zc4wra6Nj5yxZJID8WM4fuqLwJXhrA3S0luGglxMqXLMPpaNVjpb2j9n5svoG9oJCpbNIxlgT+MttW3MjDssfz+LFSUncqSnFQy5yUGpQ3zRHyVPVMp6ax194R0Z2S80pIToj637xMbr3KW2HK4hXmJ8VxRmhPpoZhx/OdHLgnLRcyXF2RwMMDKnu7+IU519Vsppwkqy/GHiaqypdrNVctySU4IXuMvEzxri7LC0ptlWX46x9rP0j90/sqeupbe0ccYEywW+MOkrqWHxo4+rl2ZF+mhmAhbXhB4ZY9V9JhQsMAfJlsOe8s4r7H8fswr91X2BJDnr3X3kBgvFIfovAITmyzwh8kLh92sKMhgUda8SA/FRFiJy6nsCaCks9bd410/3v5UTfDYpykMuvuH2NHQbhddMQAkJcRREmBlT12LXXXLBJ8F/jDYWtvKsEe5doXl943X8oL08zZrGxge4VhbrwV+E3QW+MNgy+EWMlISuDRI1281c9+y/AyOn6ey52hrLx61A7sm+Czwh5ivjHNDeR6Jlqc1juUF6XjOU9ljFT0mVAKKRCKyUUSqRaRWRO4dZ3m2iGwSkf0i8oaIrPFbliUiT4jIYRF5U0Qqg/kCZruDzV24uwcsv2/exnfRdF9wH0+tuwcRbwMxY4LpvIFfROKBB4EbgNXA7SKyesxq9wF7VXWoq6DjAAAToElEQVQt8AngAb9lDwDPqOpK4CLgzWAMfK540bmo+juXW37fvGWp07PnyCR5/lp3D0XZ80hJtBP+THAFMuNfB9Sqar2qDgKPATePWWc18AKAqh4GSkSkQEQygQ3AfzvLBlX1TNBGPwdsqW5hbVEmeRnJkR6KmUV8lT2T1fLb5RZNqAQS+BcBJ/xuNzr3+dsH3AogIuuAYqAIWAq0AD8UkT0i8gMRGbcfsYjcKSI7RWRnS0vLFF/G7NTRO8ie4x120pYZV3l+OjUTpHpGPEp9q1X0mNAI1tHG+4EsEdkL3APsAUbwNoG7FPieql4C9ALnHCMAUNWHVbVCVSvy8qIjLfJyTQsexco4zbjKCzI41tY7bmVPY8dZBoc9FvhNSAQS+JuAxX63i5z7Rqlql6reoaoX483x5wH1eL8dNKrq686qT+DdEcSELYfduNKSuKgoK9JDMbNQeb63sqfeacTmzyp6TCgFEvh3AOUislREkoDbgM3+KziVO74m5p8BXnZ2BqeAEyKywll2HXAoSGOf1UY8yktHWnjn8jziYuDCDmbqlhd4K3vGa90wGvjzMsI6JhMbztuPX1WHReRu4FkgHnhEVQ+KyF3O8oeAVcCjIqLAQeDTfk9xD/AzZ8dQD9wR5NcwK+1rPEPH2SGusTJOM4GS3FTiJ7gaV627h9z0ZDJTEyMwMhPtAroQi6o+DTw95r6H/H7fBiyf4LF7gYoZjHFOevGwmziBDeXRf/1OMz3JCfGUuFLHnfHXuHsotzSPCRE7lTREXqh2c1lxNlmpobuMn5n7yvMzzpnxqyp1bmvOZkLHAn8IuLv6qWrqsjJOc17LC9JpaOtlYPityh539wDdA8MW+E3IWOAPgRePeM9DsIuqm/NZVpBxTmWPVfSYULPAHwIvVrspmJ/MqoVWkWEmt9y5Gpd/6wYL/CbULPAH2dCIh1eOtHLtinxErIzTTG5pbhrxcfK2Zm217h4ykhPItzYfJkQs8AfZzoYOugeGrRunCUhyQjzFrtRzZvxl+ek2cTAhY4E/yF6sdpMYL1y1zMo4TWCW52e8rWdPrV1u0YSYBf4g21LtZt3SHNKTAzpFwhjKC9I51naWgeEROvuGaOkesMBvQsqiUxA1dpzlyOkePlSx+PwrG+MoL8hgxKMcbe2ld8Bb1mntmE0oWeAPohernTJOy++bKfCdoXvkdA99g8Pe+wos8JvQscAfRFsOu1mSk0pp7riXHDBmXKV5acQJ1J7upm9ohKSEOIqyUyM9LBPFLMcfJE/sauSFajfXry6wagwzJd6ePWkcOd1DrbuHUqfE05hQsRl/EPxq5wm+/Ov9XFWWyxffu+L8DzBmjPKCdGrc3QyOeOz6DSbkbMY/Q790gv7Vy3L5wScr7MLYZlrK8zNoaDtLY0efVfSYkLMZ/wz8cscJ/vZJb9D//ics6JvpKy9IZ8SjgLVqMKFngX+aHt9xnHufPMA7yvN4+OOXWdA3M1Ke/1ZfJwv8JtQs8E/DY294g/6G5Rb0TXD4KnvA27/HmFCywD9Fv3jjOH/35AHeuTyP/7Kgb4IkJTGeYlcaqkpygn2mTGhZ4J+Cn79+nPs2HeCaFXk89DEL+ia4PllZjJPmNyakLPAH6GevH+PvN1Vx7Yo8Hvr4ZTYrM0H3qauWRnoIJkZY4A/AT7cf4x+equJdK/P53scutaBvjJnTrI7/PH6yrYF/eKqK6yzoG2OihAX+Sfx4WwP/+zcHefeqfL5rQd8YEyUs8E/gx9sa+IoT9B/8qAV9Y0z0sBz/OB59rYGvbj7Iu1cV8N2PXkpSgu0fjTHRwwL/GD/cepR/+u0h3rO6gAc/YkHfGBN9LKr5efS1Bv7pt4d47wUW9I0x0csim2NoxMM/P/0mG5bn8R+3W9A3xkSvgKKbiGwUkWoRqRWRe8dZni0im0Rkv4i8ISJr/JY1iMgBEdkrIjuDOfhgqjndw8Cwhz+/rMiCvjEmqp03xy8i8cCDwHuARmCHiGxW1UN+q90H7FXVW0RkpbP+dX7Lr1XV1iCOO+iqmjsBWFM4P8IjMcaY0ApkarsOqFXVelUdBB4Dbh6zzmrgBQBVPQyUiEhBUEcaYlVNnaQnJ1Diss6IxpjoFkjgXwSc8Lvd6Nznbx9wK4CIrAOKgSJnmQLPi8guEblzoo2IyJ0islNEdra0tAQ6/qCpaupkdeF84uxap8aYKBesZPb9QJaI7AXuAfYAI86yq1X1YuAG4HMismG8J1DVh1W1QlUr8vLygjSswIx4lEMnu1hTmBnW7RpjTCQEUsffBCz2u13k3DdKVbuAOwBERICjQL2zrMn51y0im/Cmjl6e8ciDqL6lh/4hD2sWWX7fGBP9Apnx7wDKRWSpiCQBtwGb/VcQkSxnGcBngJdVtUtE0kQkw1knDbgeqAre8IPjQJNzYHeRzfiNMdHvvDN+VR0WkbuBZ4F44BFVPSgidznLHwJWAY+KiAIHgU87Dy8ANnm/BJAA/FxVnwn+y5iZqqYuUhLjKMuza50aY6JfQC0bVPVp4Okx9z3k9/s2YPk4j6sHLprhGEOuqrmT1QvnE28Hdo0xMSDmz1TyeJRDzV2W5jHGxIyYD/wNbb30DAxbRY8xJmbEfOCvau4C7MCuMSZ2xHzgP9jUSVJ8HOUFdmDXGBMbYj7wVzV3snJhBonxMf9WGGNiRExHO1WlqskO7BpjYktMB/7Gjj46+4bswK4xJqbEdOCvGj1j11o1GGNiR2wH/uZOEuKE5QUZkR6KMcaETUwH/gNNXSwvyCAlMT7SQzHGmLCJ2cCvqhxs6rQ0jzEm5sRs4D/V1U9b76BV9BhjYk7MBv4Djd4DuxdYRY8xJsbEbOCvau4iTmD1Qkv1GGNiS8wG/oNNnSzLT2dekh3YNcbElpgN/FXNnXbiljEmJsVk4Hd393O6a4AL7MCuMSYGxWTgP9jkbcV8oQV+Y0wMisnA72vVsLrQDuwaY2JPbAb+5k5Kc9NITw7oksPGGBNVYjPwWytmY0wMi7nA3947SNOZPmvVYIyJWTEX+A82O62YrZTTGBOjYi7wVzkVPdaqwRgTq2Iw8HeyJCeVzNTESA/FGGMiIvYCf7O1YjbGxLaYCvydfUMcaztraR5jTEyLqcB/qNmb37dSTmNMLIupwD96cXU7Y9cYE8MCCvwislFEqkWkVkTuHWd5tohsEpH9IvKGiKwZszxeRPaIyO+CNfDpqGrupDAzBVd6ciSHYYwxEXXewC8i8cCDwA3AauB2EVk9ZrX7gL2quhb4BPDAmOWfB96c+XBnpqqp0zpyGmNiXiAz/nVArarWq+og8Bhw85h1VgMvAKjqYaBERAoARKQIeB/wg6CNehp6B4apb+21E7eMMTEvkMC/CDjhd7vRuc/fPuBWABFZBxQDRc6ybwNfBjyTbURE7hSRnSKys6WlJYBhTc2hk12owoVFlt83xsS2YB3cvR/IEpG9wD3AHmBERN4PuFV11/meQFUfVtUKVa3Iy8sL0rDe8taBXZvxG2NiWyB9iZuAxX63i5z7RqlqF3AHgIgIcBSoBz4M3CQiNwIpwHwR+amqfiwIY5+SqqYu8jKSyZ+fEu5NG2PMrBLIjH8HUC4iS0UkCbgN2Oy/gohkOcsAPgO8rKpdqvp3qlqkqiXO416IRNAH74zfrrhljDEBBH5VHQbuBp7FW5nzS1U9KCJ3ichdzmqrgCoRqcZb/fP5UA14OvoGR6hxd1v9vjHGEFiqB1V9Gnh6zH0P+f2+DVh+nud4EXhxyiMMgsOnuvAoVsppjDHEyJm7VdaqwRhjRsVG4G/sJCcticJMO7BrjDGxEfibO7mgcD7egiNjjIltUR/4B4ZHOHK629I8xhjjiPrAX3O6h6ERtRO3jDHGEfWB/4Bzxq7V8BtjjFfUB/6qpk4yUhJYnDMv0kMxxphZIfoDf3MXawoz7cCuMcY4ojrwD414ePNkFxcWWZrHGGN8ojrw17p7GBz2cIG1ajDGmFFRHfhHWzHbgV1jjBkV1YH/YHMXaUnxLHWlRXooxhgza0R14K9q6uSCwkzi4uzArjHG+ERt4B/xKAebu7hgkeX3jTHGX9QG/qOtPfQNjdgZu8YYM0bUBv6qJmvFbIwx44nawH+gqZOUxDjK8uzArjHG+IvawF/V1MmqhfNJiI/al2iMMdMSlVHR41EOOa0ajDHGvF1UBv7j7WfpHhhmjVX0GGPMOaIy8B+wM3aNMWZCURn4q5o7SYqPozw/I9JDMcaYWScqA//Bpi5WLMggKSEqX54xxsxI1EVGVaWqudPSPMYYM4GoC/yNHX2cOTtkB3aNMWYCURf4DzY7B3atlNMYY8YVdYG/qqmLhDhhxQI7sGuMMeOJvsDf3El5QQYpifGRHooxxsxKAQV+EdkoItUiUisi946zPFtENonIfhF5Q0TWOPenOLf3ichBEfmnYL8Af6pKVVMna+xSi8YYM6HzBn4RiQceBG4AVgO3i8jqMavdB+xV1bXAJ4AHnPsHgHep6kXAxcBGEVkfrMGPdbprgNaeQavoMcaYSQQy418H1KpqvaoOAo8BN49ZZzXwAoCqHgZKRKRAvXqcdRKdHw3O0M/11jV2bcZvjDETCSTwLwJO+N1udO7ztw+4FUBE1gHFQJFzO15E9gJu4DlVfX28jYjInSKyU0R2trS0TO1VOKqaO4kTWLXQAr8xxkwkWAd37weynAB/D7AHGAFQ1RFVvRjvjmCdL/8/lqo+rKoVqlqRl5c3rUFUNXVSlpdOalLCtB5vjDGxIJAI2QQs9rtd5Nw3SlW7gDsARESAo0D9mHXOiMgWYCNQNYMxT6iqqYvKMlcontoYY6JGIDP+HUC5iCwVkSTgNmCz/woikuUsA/gM8LKqdolInohkOevMA94DHA7e8N8yOOzh6vJc3rl8et8WjDEmVpx3xq+qwyJyN/AsEA88oqoHReQuZ/lDwCrgURFR4CDwaefhC5374/HuZH6pqr8LwesgKSGOf/2Li0Lx1MYYE1VENWRFNtNWUVGhO3fujPQwjDFmzhCRXapaEci6UXfmrjHGmMlZ4DfGmBhjgd8YY2KMBX5jjIkxFviNMSbGWOA3xpgYY4HfGGNizKys4xeRFuDYNB+eC7QGcTjBZuObGRvfzNj4ZmY2j69YVQNqXTArA/9MiMjOQE9iiAQb38zY+GbGxjczs318gbJUjzHGxBgL/MYYE2OiMfA/HOkBnIeNb2ZsfDNj45uZ2T6+gERdjt8YY8zkonHGb4wxZhIW+I0xJsbMycAvIhtFpFpEakXk3nGWi4h8x1m+X0QuDfP4FovIFhE5JCIHReTz46xzjYh0ishe5+crYR5jg4gccLZ9zsUPIvkeisgKv/dlr4h0icgXxqwT1vdPRB4REbeIVPndlyMiz4lIjfNv9gSPnfTzGsLxfVNEDjv/f5t8V8Mb57GTfhZCOL5/FJEmv//DGyd4bKTev8f9xtbgXFN8vMeG/P0LOlWdUz94rwJWB5QCScA+YPWYdW4E/gAIsB54PcxjXAhc6vyeARwZZ4zXAL+L4PvYAOROsjyi7+GY/+9TeE9Oidj7B2wALgWq/O77BnCv8/u9wNcnGP+kn9cQju96IMH5/evjjS+Qz0IIx/ePwBcD+P+PyPs3Zvm/AV+J1PsX7J+5OONfB9Sqar2qDgKPATePWedm4MfqtR3IEpGF4Rqgqp5U1d3O793Am8CicG0/SCL6Hvq5DqhT1emeyR0Uqvoy0D7m7puBR53fHwU+MM5DA/m8hmR8qvpHVR12bm4HioK93UBN8P4FImLvn4+ICPAh4BfB3m6kzMXAvwg44Xe7kXODaiDrhIWIlACXAK+Ps/hK52v4H0TkgrAODBR4XkR2icid4yyfLe/hbUz8BxfJ9w+gQFVPOr+fAgrGWWe2vI9/ifcb3HjO91kIpXuc/8NHJkiVzYb37x3AaVWtmWB5JN+/aZmLgX/OEJF04NfAF1S1a8zi3cASVV0L/AfwVJiHd7WqXgzcAHxORDaEefvnJSJJwE3Ar8ZZHOn3723U+51/VtZGi8jfA8PAzyZYJVKfhe/hTeFcDJzEm06ZjW5n8tn+rP9bGmsuBv4mYLHf7SLnvqmuE1Iikog36P9MVZ8cu1xVu1S1x/n9aSBRRHLDNT5VbXL+dQOb8H6l9hfx9xDvH9JuVT09dkGk3z/HaV/6y/nXPc46EX0fReRTwPuBjzo7p3ME8FkICVU9raojquoBvj/BdiP9/iUAtwKPT7ROpN6/mZiLgX8HUC4iS50Z4W3A5jHrbAY+4VSmrAc6/b6Sh5yTE/xv4E1V/dYE6yxw1kNE1uH9v2gL0/jSRCTD9zveg4BVY1aL6HvomHCmFcn3z89m4JPO758EfjPOOoF8XkNCRDYCXwZuUtWzE6wTyGchVOPzP2Z0ywTbjdj753g3cFhVG8dbGMn3b0YifXR5Oj94K06O4D3a//fOfXcBdzm/C/Cgs/wAUBHm8V2N92v/fmCv83PjmDHeDRzEW6WwHbgyjOMrdba7zxnDbHwP0/AG8ky/+yL2/uHdAZ0EhvDmmT8NuIA/ATXA80COs24h8PRkn9cwja8Wb37c9xl8aOz4JvoshGl8P3E+W/vxBvOFs+n9c+7/ke8z57du2N+/YP9YywZjjIkxczHVY4wxZgYs8BtjTIyxwG+MMTHGAr8xxsQYC/zGGBNjLPAbY0yMscBvjDEx5v8DD4zKq+qWgTkAAAAASUVORK5CYII=",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1135c4fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(len(eval_acces)), eval_acces)\n",
    "plt.title('test acc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到我们的三层网络在训练集上能够达到 99.9% 的准确率，测试集上能够达到 98.20% 的准确率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**小练习：看一看上面的训练过程，看一下准确率是怎么计算出来的，特别注意 max 这个函数**\n",
    "\n",
    "**自己重新实现一个新的网络，试试改变隐藏层的数目和激活函数，看看有什么新的结果**"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "07b2cea26089ea0ebec7cc3a83022d31e9f80d0db55f432f89380becb3d80933"
  },
  "kernelspec": {
   "display_name": "mx",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
